[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:06:42 dist-test-slave-03rn krb5kdc[19077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:06:42 dist-test-slave-03rn krb5kdc[19077](info): set up 2 sockets
Sep 26 02:06:42 dist-test-slave-03rn krb5kdc[19077](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 02:06:43.317335 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:06:43 dist-test-slave-03rn krb5kdc[19077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:06:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:06:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:06:43.522583 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:35465
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:38757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:35465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:06:43.853230 19090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:06:43.853576 19090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:06:43.853709 19090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:06:43.861126 19090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:06:43.861258 19090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:06:43.861302 19090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:06:43.861436 19090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:06:43.861505 19090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:06:43.871618 19090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:38757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:35465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:35465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:06:43.873040 19090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:06:43.874421 19090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:06:43.886799 19098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:43.886809 19096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:43.887276 19095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:06:43.887712 19090 server_base.cc:1034] running on GCE node
I20260926 02:06:43.888669 19090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:06:43.890352 19090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:06:43.891599 19090 hybrid_clock.cc:648] HybridClock initialized: now 1790388403891542 us; error 76 us; skew 500 ppm
I20260926 02:06:43.891985 19090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:06:43 dist-test-slave-03rn krb5kdc[19077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388403, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:06:43.895416 19090 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:06:43.897156 19090 webserver.cc:460] Webserver started at http://127.18.159.190:41125/ using document root <none> and password file <none>
I20260926 02:06:43.897907 19090 fs_manager.cc:359] Metadata directory not provided
I20260926 02:06:43.898064 19090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:06:43.898378 19090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:06:43.901572 19090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "92dcc4b4aa1441878addd4698cd8d32d"
format_stamp: "Formatted at 2026-09-26 02:06:43 on dist-test-slave-03rn"
server_key: "4b52e09917862fe22c883450c86c095c"
server_key_iv: "5e5b13f1526afba5ed2f45a03cb78609"
server_key_version: "encryptionkey@0"
I20260926 02:06:43.902381 19090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "92dcc4b4aa1441878addd4698cd8d32d"
format_stamp: "Formatted at 2026-09-26 02:06:43 on dist-test-slave-03rn"
server_key: "4b52e09917862fe22c883450c86c095c"
server_key_iv: "5e5b13f1526afba5ed2f45a03cb78609"
server_key_version: "encryptionkey@0"
I20260926 02:06:43.907989 19090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 02:06:43.912030 19105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:43.913839 19090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:06:43.914089 19090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "92dcc4b4aa1441878addd4698cd8d32d"
format_stamp: "Formatted at 2026-09-26 02:06:43 on dist-test-slave-03rn"
server_key: "4b52e09917862fe22c883450c86c095c"
server_key_iv: "5e5b13f1526afba5ed2f45a03cb78609"
server_key_version: "encryptionkey@0"
I20260926 02:06:43.914318 19090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:06:43.934950 19090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:06:43.940021 19090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:06:43.940366 19090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:06:43.960412 19090 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:35465
I20260926 02:06:43.960420 19156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:35465 every 8 connection(s)
I20260926 02:06:43.961943 19090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:06:43.965863 19157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:06:43.971120 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 19090
I20260926 02:06:43.971446 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:06:43.973546 19070 external_mini_cluster.cc:1428] Setting key 6178cab33dac05c806a21e7ae2462376
I20260926 02:06:43.976888 19157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d: Bootstrap starting.
I20260926 02:06:43.981006 19157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d: Neither blocks nor log segments found. Creating new log.
I20260926 02:06:43.982087 19157 log.cc:824] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:06:43.985146 19157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d: No bootstrap required, opened a new log
I20260926 02:06:43.994395 19157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92dcc4b4aa1441878addd4698cd8d32d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35465 } }
I20260926 02:06:43.994854 19157 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:06:43.994994 19157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92dcc4b4aa1441878addd4698cd8d32d, State: Initialized, Role: FOLLOWER
I20260926 02:06:43.995622 19157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92dcc4b4aa1441878addd4698cd8d32d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35465 } }
I20260926 02:06:43.995914 19157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:06:43.996071 19157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:06:43.996282 19157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:06:43.998519 19157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92dcc4b4aa1441878addd4698cd8d32d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35465 } }
I20260926 02:06:43.999112 19157 leader_election.cc:302] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92dcc4b4aa1441878addd4698cd8d32d; no voters:
I20260926 02:06:43.999810 19157 leader_election.cc:288] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:06:43.999992 19162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:06:44.001173 19162 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d [term 1 LEADER]: Becoming Leader. State: Replica: 92dcc4b4aa1441878addd4698cd8d32d, State: Running, Role: LEADER
I20260926 02:06:44.001781 19162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92dcc4b4aa1441878addd4698cd8d32d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35465 } }
I20260926 02:06:44.002334 19157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:06:44.004971 19163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92dcc4b4aa1441878addd4698cd8d32d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92dcc4b4aa1441878addd4698cd8d32d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35465 } } }
I20260926 02:06:44.005458 19163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d [sys.catalog]: This master's current role is: LEADER
Sep 26 02:06:44 dist-test-slave-03rn krb5kdc[19077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:06:44.009727 19164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92dcc4b4aa1441878addd4698cd8d32d. Latest consensus state: current_term: 1 leader_uuid: "92dcc4b4aa1441878addd4698cd8d32d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92dcc4b4aa1441878addd4698cd8d32d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35465 } } }
I20260926 02:06:44.010071 19164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d [sys.catalog]: This master's current role is: LEADER
I20260926 02:06:44.012842 19167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:06:44.018730 19167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:06:44.020427 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:43.977723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.18.159.190:35465)
0926 02:06:43.978682 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:43.978702 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:43.979875 (+ 1173us) server_negotiation.cc:407] Connection header received
0926 02:06:43.990149 (+ 10274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:43.990177 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:43.990710 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:43.991229 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:43.994724 (+ 3495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:43.996629 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:43.998483 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:43.999435 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:43.999654 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:44.007947 (+ 8293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:44.007964 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:44.007995 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:44.008067 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:44.013663 (+ 5596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:44.015009 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:44.015017 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:44.015023 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:44.015251 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:44.016002 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:44.016007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:44.016010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:44.016834 (+ 824us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:44.017134 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:44.018401 (+ 1267us) server_negotiation.cc:299] Negotiation successful
0926 02:06:44.018866 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":424,"threads_started":1}
I20260926 02:06:44.029312 19167 catalog_manager.cc:1347] Generated new cluster ID: f2a78964296c468e97b50501adc889b4
I20260926 02:06:44.029443 19167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:06:44.051304 19167 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:06:44.052788 19167 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:06:44.070436 19167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92dcc4b4aa1441878addd4698cd8d32d: Generated new TSK 0
I20260926 02:06:44.071417 19167 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.159.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
I20260926 02:06:44.119712 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35465
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:38757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:06:44.474385 19183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:06:44.474699 19183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:06:44.474794 19183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:06:44.482558 19183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:06:44.482692 19183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:06:44.482811 19183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:06:44.493470 19183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:38757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35465
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:06:44.494764 19183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:06:44.496260 19183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:06:44.510862 19191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:44.510951 19188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:44.510902 19189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:06:44.511969 19183 server_base.cc:1034] running on GCE node
I20260926 02:06:44.512718 19183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:06:44.513860 19183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:06:44.515123 19183 hybrid_clock.cc:648] HybridClock initialized: now 1790388404515059 us; error 79 us; skew 500 ppm
I20260926 02:06:44.515491 19183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:06:44 dist-test-slave-03rn krb5kdc[19077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388404, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:06:44.519078 19183 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:06:44.520967 19183 webserver.cc:460] Webserver started at http://127.18.159.129:36259/ using document root <none> and password file <none>
I20260926 02:06:44.521703 19183 fs_manager.cc:359] Metadata directory not provided
I20260926 02:06:44.521862 19183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:06:44.522184 19183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:06:44.525265 19183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "86dd6f73a7a14a0aa5a528a130450fcf"
format_stamp: "Formatted at 2026-09-26 02:06:44 on dist-test-slave-03rn"
server_key: "d6aba18cb29d583c30c4f4bb963b473a"
server_key_iv: "db7de17452e31426536c258e79755bfd"
server_key_version: "encryptionkey@0"
I20260926 02:06:44.526081 19183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "86dd6f73a7a14a0aa5a528a130450fcf"
format_stamp: "Formatted at 2026-09-26 02:06:44 on dist-test-slave-03rn"
server_key: "d6aba18cb29d583c30c4f4bb963b473a"
server_key_iv: "db7de17452e31426536c258e79755bfd"
server_key_version: "encryptionkey@0"
I20260926 02:06:44.531688 19183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:06:44.536777 19198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:44.538458 19183 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:06:44.538694 19183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "86dd6f73a7a14a0aa5a528a130450fcf"
format_stamp: "Formatted at 2026-09-26 02:06:44 on dist-test-slave-03rn"
server_key: "d6aba18cb29d583c30c4f4bb963b473a"
server_key_iv: "db7de17452e31426536c258e79755bfd"
server_key_version: "encryptionkey@0"
I20260926 02:06:44.538944 19183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:06:44.568531 19183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:06:44.574618 19183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:06:44.574947 19183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:06:44.576208 19183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:06:44.578270 19183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:06:44.578385 19183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:44.578531 19183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:06:44.578604 19183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:44.616420 19183 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:33507
I20260926 02:06:44.616459 19310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:33507 every 8 connection(s)
I20260926 02:06:44.617806 19183 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:06:44.619050 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 19183
I20260926 02:06:44.619292 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:06:44.619911 19070 external_mini_cluster.cc:1428] Setting key fc818ba698b772161aeede91bc116d10
Sep 26 02:06:44 dist-test-slave-03rn krb5kdc[19077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388404, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.159.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.130@KRBTEST.COM" created.
I20260926 02:06:44.641498 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:44.620701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:38763 (local address 127.18.159.190:35465)
0926 02:06:44.621564 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:44.621570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:44.622707 (+ 1137us) server_negotiation.cc:407] Connection header received
0926 02:06:44.624703 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:44.624712 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:44.624808 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:44.624931 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:44.626777 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:44.627506 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:44.630301 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:44.630724 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:44.630832 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:44.635259 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:44.635264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:44.635268 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:44.635303 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:44.638026 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:44.638892 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:44.638898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:44.638900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:44.639030 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:44.639518 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:44.639555 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:44.639557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:44.639818 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:44.639980 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:44.641036 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0926 02:06:44.641245 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":606,"threads_started":1}
I20260926 02:06:44.643764 19314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:44.621434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35465 (local address 127.18.159.129:38763)
0926 02:06:44.622500 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:44.622563 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:44.624398 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:44.625157 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:44.625166 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:44.625743 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:06:44.626544 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:44.626572 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:44.627683 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:44.627692 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:44.630111 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:44.630129 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:44.630985 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:44.630994 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:44.631349 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:44.632141 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:06:44.632214 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:06:44.634960 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:06:44.638209 (+ 3249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:44.638221 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:44.638243 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:44.638734 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:44.639155 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:44.639160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:44.639164 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:44.639395 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:44.639966 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:06:44.639974 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:06:44.640522 (+ 548us) client_negotiation.cc:769] Sending connection context
0926 02:06:44.640874 (+ 352us) client_negotiation.cc:240] Negotiation successful
0926 02:06:44.641506 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":550,"threads_started":1}
I20260926 02:06:44.645573 19311 heartbeater.cc:347] Connected to a master server at 127.18.159.190:35465
I20260926 02:06:44.645901 19311 heartbeater.cc:464] Registering TS with master...
I20260926 02:06:44.646626 19311 heartbeater.cc:511] Master 127.18.159.190:35465 requested a full tablet report, sending...
I20260926 02:06:44.648993 19122 ts_manager.cc:194] Registered new tserver with Master: 86dd6f73a7a14a0aa5a528a130450fcf (127.18.159.129:33507)
I20260926 02:06:44.651221 19122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:38763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
I20260926 02:06:44.664433 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35465
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:38757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:06:45.021466 19317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:06:45.021811 19317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:06:45.021987 19317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:06:45.030054 19317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:06:45.030210 19317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:06:45.030396 19317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:06:45.041182 19317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:38757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35465
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:06:45.042630 19317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:06:45.044126 19317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:06:45.059171 19325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:45.059064 19322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:45.059082 19323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:06:45.060492 19317 server_base.cc:1034] running on GCE node
I20260926 02:06:45.061194 19317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:06:45.062237 19317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:06:45.063493 19317 hybrid_clock.cc:648] HybridClock initialized: now 1790388405063443 us; error 38 us; skew 500 ppm
I20260926 02:06:45.063855 19317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:06:45 dist-test-slave-03rn krb5kdc[19077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388405, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:06:45.067389 19317 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:06:45.069624 19317 webserver.cc:460] Webserver started at http://127.18.159.130:35709/ using document root <none> and password file <none>
I20260926 02:06:45.070461 19317 fs_manager.cc:359] Metadata directory not provided
I20260926 02:06:45.070580 19317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:06:45.070943 19317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:06:45.074119 19317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "562886a1af5243d89767a9ada4efce76"
format_stamp: "Formatted at 2026-09-26 02:06:45 on dist-test-slave-03rn"
server_key: "b0ac9e3260cb27278da82447175e302e"
server_key_iv: "e989d18359ddf854e3c4de41abc4a022"
server_key_version: "encryptionkey@0"
I20260926 02:06:45.074965 19317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "562886a1af5243d89767a9ada4efce76"
format_stamp: "Formatted at 2026-09-26 02:06:45 on dist-test-slave-03rn"
server_key: "b0ac9e3260cb27278da82447175e302e"
server_key_iv: "e989d18359ddf854e3c4de41abc4a022"
server_key_version: "encryptionkey@0"
I20260926 02:06:45.080904 19317 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 02:06:45.085106 19332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:45.086922 19317 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:06:45.087200 19317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "562886a1af5243d89767a9ada4efce76"
format_stamp: "Formatted at 2026-09-26 02:06:45 on dist-test-slave-03rn"
server_key: "b0ac9e3260cb27278da82447175e302e"
server_key_iv: "e989d18359ddf854e3c4de41abc4a022"
server_key_version: "encryptionkey@0"
I20260926 02:06:45.087477 19317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:06:45.116714 19317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:06:45.122292 19317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:06:45.122632 19317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:06:45.124207 19317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:06:45.125815 19317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:06:45.125902 19317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:45.126071 19317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:06:45.126132 19317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:45.168107 19317 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:35939
I20260926 02:06:45.168148 19444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:35939 every 8 connection(s)
I20260926 02:06:45.169593 19317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:06:45.174371 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 19317
I20260926 02:06:45.174584 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:06:45.175009 19070 external_mini_cluster.cc:1428] Setting key 9a86b4184ae10d0da7820e6d3d741a04
Sep 26 02:06:45 dist-test-slave-03rn krb5kdc[19077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388405, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.159.131@KRBTEST.COM; defaulting to no policy
I20260926 02:06:45.196609 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:45.172532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:50341 (local address 127.18.159.190:35465)
0926 02:06:45.173008 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:45.173015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:45.174092 (+ 1077us) server_negotiation.cc:407] Connection header received
0926 02:06:45.176489 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:45.176495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:45.176589 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:45.176735 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:45.179161 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:45.179889 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:45.182868 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:45.183667 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:45.183835 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:45.187781 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:45.187791 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:45.187799 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:45.187848 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:45.192444 (+ 4596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:45.193713 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:45.193721 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:45.193726 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:45.193914 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:45.194516 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:45.194523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:45.194526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:45.194966 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:45.195128 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:45.196032 (+ 904us) server_negotiation.cc:299] Negotiation successful
0926 02:06:45.196331 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":231,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.131@KRBTEST.COM" created.
I20260926 02:06:45.197948 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:45.173168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35465 (local address 127.18.159.130:50341)
0926 02:06:45.173900 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:45.173944 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:45.176109 (+ 2165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:45.177031 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:45.177041 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:45.177920 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:06:45.178893 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:45.178926 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:45.180088 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:45.180097 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:45.182658 (+ 2561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:45.182686 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:45.183945 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:45.183951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:45.184256 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:45.185002 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:06:45.185025 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:06:45.187488 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:06:45.192803 (+ 5315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:45.192813 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:45.192880 (+ 67us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:45.193460 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:45.194105 (+ 645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:45.194109 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:45.194112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:45.194360 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:45.195148 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:06:45.195156 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:06:45.195546 (+ 390us) client_negotiation.cc:769] Sending connection context
0926 02:06:45.195928 (+ 382us) client_negotiation.cc:240] Negotiation successful
0926 02:06:45.196462 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":283,"threads_started":1}
I20260926 02:06:45.199735 19445 heartbeater.cc:347] Connected to a master server at 127.18.159.190:35465
I20260926 02:06:45.200133 19445 heartbeater.cc:464] Registering TS with master...
I20260926 02:06:45.201123 19445 heartbeater.cc:511] Master 127.18.159.190:35465 requested a full tablet report, sending...
I20260926 02:06:45.202919 19122 ts_manager.cc:194] Registered new tserver with Master: 562886a1af5243d89767a9ada4efce76 (127.18.159.130:35939)
I20260926 02:06:45.204249 19122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:50341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
I20260926 02:06:45.223201 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35465
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:38757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:06:45.583264 19451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:06:45.583580 19451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:06:45.583735 19451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:06:45.592303 19451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:06:45.592466 19451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:06:45.592615 19451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:06:45.603346 19451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:38757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35465
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:06:45.604785 19451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:06:45.606247 19451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:06:45.622798 19456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:45.622845 19457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:45.623327 19459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:06:45.624719 19451 server_base.cc:1034] running on GCE node
I20260926 02:06:45.625491 19451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:06:45.626565 19451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:06:45.627847 19451 hybrid_clock.cc:648] HybridClock initialized: now 1790388405627780 us; error 55 us; skew 500 ppm
I20260926 02:06:45.628324 19451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:06:45 dist-test-slave-03rn krb5kdc[19077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388405, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:06:45.632619 19451 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:06:45.634558 19451 webserver.cc:460] Webserver started at http://127.18.159.131:36633/ using document root <none> and password file <none>
I20260926 02:06:45.635334 19451 fs_manager.cc:359] Metadata directory not provided
I20260926 02:06:45.635492 19451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:06:45.635843 19451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:06:45.639096 19451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "b0e83d637e1a4997834a63faaac0302e"
format_stamp: "Formatted at 2026-09-26 02:06:45 on dist-test-slave-03rn"
server_key: "5314e351bbf379919d9d55c3d170851c"
server_key_iv: "b9f406314954cd9771fd5832d5488482"
server_key_version: "encryptionkey@0"
I20260926 02:06:45.639926 19451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "b0e83d637e1a4997834a63faaac0302e"
format_stamp: "Formatted at 2026-09-26 02:06:45 on dist-test-slave-03rn"
server_key: "5314e351bbf379919d9d55c3d170851c"
server_key_iv: "b9f406314954cd9771fd5832d5488482"
server_key_version: "encryptionkey@0"
I20260926 02:06:45.646674 19451 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260926 02:06:45.651201 19466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:45.652874 19451 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:06:45.653090 19451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "b0e83d637e1a4997834a63faaac0302e"
format_stamp: "Formatted at 2026-09-26 02:06:45 on dist-test-slave-03rn"
server_key: "5314e351bbf379919d9d55c3d170851c"
server_key_iv: "b9f406314954cd9771fd5832d5488482"
server_key_version: "encryptionkey@0"
I20260926 02:06:45.653326 19451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:06:45.654681 19311 heartbeater.cc:503] Master 127.18.159.190:35465 was elected leader, sending a full tablet report...
I20260926 02:06:45.670166 19451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:06:45.675102 19451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:06:45.675383 19451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:06:45.676504 19451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:06:45.678404 19451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:06:45.678499 19451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260926 02:06:45.678598 19451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:06:45.678643 19451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:45.725232 19451 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:37715
I20260926 02:06:45.725286 19578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:37715 every 8 connection(s)
I20260926 02:06:45.726673 19451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:06:45.733371 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 19451
I20260926 02:06:45.733578 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:06:45.734157 19070 external_mini_cluster.cc:1428] Setting key 793ec97b91d953bbb7b77fe9fb5aaf36
Sep 26 02:06:45 dist-test-slave-03rn krb5kdc[19077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388405, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:06:45.754379 19581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:45.731422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:45057 (local address 127.18.159.190:35465)
0926 02:06:45.731992 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:45.731997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:45.733250 (+ 1253us) server_negotiation.cc:407] Connection header received
0926 02:06:45.735675 (+ 2425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:45.735681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:45.735769 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:45.735934 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:45.738543 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:45.739427 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:45.741315 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:45.741733 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:45.741873 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:45.747508 (+ 5635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:45.747514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:45.747517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:45.747559 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:45.750535 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:45.751518 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:45.751531 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:45.751533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:45.751667 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:45.752178 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:45.752189 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:45.752191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:45.752478 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:45.752655 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:45.753711 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0926 02:06:45.753955 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":331,"threads_started":1}
I20260926 02:06:45.755602 19582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:45.732096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35465 (local address 127.18.159.131:45057)
0926 02:06:45.733016 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:45.733082 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:45.735390 (+ 2308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:45.736311 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:45.736328 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:45.737480 (+ 1152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:06:45.738290 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:45.738320 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:45.739636 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:45.739641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:45.741130 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:45.741146 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:45.741993 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:45.742000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:45.742260 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:45.743602 (+ 1342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:06:45.743653 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:06:45.747173 (+ 3520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:06:45.750828 (+ 3655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:45.750837 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:45.750853 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:45.751375 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:45.751792 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:45.751799 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:45.751807 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:45.752033 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:45.752624 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:06:45.752632 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:06:45.753254 (+ 622us) client_negotiation.cc:769] Sending connection context
0926 02:06:45.753611 (+ 357us) client_negotiation.cc:240] Negotiation successful
0926 02:06:45.754245 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":330,"threads_started":1}
I20260926 02:06:45.757566 19579 heartbeater.cc:347] Connected to a master server at 127.18.159.190:35465
I20260926 02:06:45.758016 19579 heartbeater.cc:464] Registering TS with master...
I20260926 02:06:45.758965 19579 heartbeater.cc:511] Master 127.18.159.190:35465 requested a full tablet report, sending...
I20260926 02:06:45.760753 19122 ts_manager.cc:194] Registered new tserver with Master: b0e83d637e1a4997834a63faaac0302e (127.18.159.131:37715)
I20260926 02:06:45.761611 19122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:45057
I20260926 02:06:45.764689 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:06:45 dist-test-slave-03rn krb5kdc[19077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388405, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 02:06:45 dist-test-slave-03rn krb5kdc[19077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388405, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:06:45.823103 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:45.800992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47682 (local address 127.18.159.129:33507)
0926 02:06:45.803275 (+ 2283us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:45.803285 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:45.803313 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:06:45.803372 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:45.803380 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:45.803592 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:45.803743 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:45.805418 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:45.806473 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:45.807794 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:45.808462 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:45.808566 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:45.812924 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:45.812940 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:45.812957 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:45.813196 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:45.818603 (+ 5407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:45.819864 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:45.819877 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:45.819881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:45.820072 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:45.820613 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:45.820618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:45.820624 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:45.821112 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:45.821457 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:45.822416 (+ 959us) server_negotiation.cc:299] Negotiation successful
0926 02:06:45.822665 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2115,"thread_start_us":1943,"threads_started":1}
W20260926 02:06:45.825246 19226 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47682
Sep 26 02:06:45 dist-test-slave-03rn krb5kdc[19077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:06:45 dist-test-slave-03rn krb5kdc[19077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:06:45.876794 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:45.859268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47690 (local address 127.18.159.129:33507)
0926 02:06:45.859591 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:45.859600 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:45.859887 (+ 287us) server_negotiation.cc:407] Connection header received
0926 02:06:45.860185 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:45.860192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:45.860297 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:45.860413 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:45.862057 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:45.862793 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:45.864419 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:45.865030 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:45.865322 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:45.869858 (+ 4536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:45.869865 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:45.869868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:45.869911 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:45.873057 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:45.874112 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:45.874117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:45.874119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:45.874269 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:45.875055 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:45.875059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:45.875062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:45.875380 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:45.875604 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:45.876096 (+ 492us) server_negotiation.cc:299] Negotiation successful
0926 02:06:45.876404 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 02:06:45.878499 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 19183
I20260926 02:06:45.897161 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:45.879724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47696 (local address 127.18.159.129:33507)
0926 02:06:45.879981 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:45.879988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:45.880196 (+ 208us) server_negotiation.cc:407] Connection header received
0926 02:06:45.880470 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:45.880482 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:45.880546 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:45.880815 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:45.882572 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:45.883512 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:45.885594 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:45.886032 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:45.886187 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:45.888831 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:45.888837 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:45.888840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:45.888879 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:45.894107 (+ 5228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:45.894999 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:45.895004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:45.895007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:45.895165 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:45.895600 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:45.895604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:45.895606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:45.895893 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:45.896156 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:45.896534 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 02:06:45.896794 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 02:06:45.897957 19306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:06:45.989178 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 19183
I20260926 02:06:46.006053 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 19317
Sep 26 02:06:46 dist-test-slave-03rn krb5kdc[19077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.130@KRBTEST.COM
I20260926 02:06:46.023876 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:46.007229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54170 (local address 127.18.159.130:35939)
0926 02:06:46.007760 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:46.007773 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:46.007801 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:06:46.007863 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:46.007870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:46.008099 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:46.008289 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:46.009511 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:46.010608 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:46.012056 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:46.012749 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:46.012864 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:46.017374 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:46.017389 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:46.017406 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:46.017456 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:46.020810 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:46.021664 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:46.021672 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:46.021674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:46.021806 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:46.022207 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:46.022210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:46.022212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:46.022596 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:46.022853 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:46.023221 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 02:06:46.023504 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":290,"threads_started":1}
I20260926 02:06:46.025350 19440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:06:46.110540 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 19317
I20260926 02:06:46.133560 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 19451
Sep 26 02:06:46 dist-test-slave-03rn krb5kdc[19077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.131@KRBTEST.COM
I20260926 02:06:46.151752 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:46.134643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42868 (local address 127.18.159.131:37715)
0926 02:06:46.135298 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:46.135305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:46.135338 (+ 33us) server_negotiation.cc:407] Connection header received
0926 02:06:46.135397 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:46.135403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:46.135625 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:46.135764 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:46.137179 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:46.138289 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:46.139818 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:46.140557 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:46.140669 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:46.145286 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:46.145302 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:46.145318 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:46.145366 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:46.148481 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:46.149315 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:46.149322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:46.149325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:46.149462 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:46.149837 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:46.149841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:46.149844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:46.150295 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:46.150541 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:46.150996 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 02:06:46.151240 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":319,"threads_started":1}
I20260926 02:06:46.153306 19574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:06:46.240342 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 19451
I20260926 02:06:46.257985 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 19090
I20260926 02:06:46.259583 19152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:06:46.364319 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 19090
2026-09-26T02:06:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4164 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:06:46 dist-test-slave-03rn krb5kdc[19601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:06:46 dist-test-slave-03rn krb5kdc[19601](info): set up 2 sockets
Sep 26 02:06:46 dist-test-slave-03rn krb5kdc[19601](info): commencing operation
krb5kdc: starting...
W20260926 02:06:47.480530 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:06:47 dist-test-slave-03rn krb5kdc[19601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:06:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:06:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:06:47.669049 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:36911
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:39597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:36911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:06:47.996279 19614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:06:47.996675 19614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:06:47.996826 19614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:06:48.004110 19614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:06:48.004314 19614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:06:48.004431 19614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:06:48.004531 19614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:06:48.004618 19614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:06:48.014559 19614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:39597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:36911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:36911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:06:48.015969 19614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:06:48.017349 19614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:06:48.029558 19620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:48.029592 19622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:48.029512 19619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:06:48.030746 19614 server_base.cc:1034] running on GCE node
I20260926 02:06:48.031581 19614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:06:48.033211 19614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:06:48.034438 19614 hybrid_clock.cc:648] HybridClock initialized: now 1790388408034390 us; error 47 us; skew 500 ppm
I20260926 02:06:48.034940 19614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:06:48 dist-test-slave-03rn krb5kdc[19601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388408, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:06:48.042235 19614 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:06:48.043924 19614 webserver.cc:460] Webserver started at http://127.18.159.190:43233/ using document root <none> and password file <none>
I20260926 02:06:48.044726 19614 fs_manager.cc:359] Metadata directory not provided
I20260926 02:06:48.044840 19614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:06:48.045102 19614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:06:48.048105 19614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "f2690709bf574d779fc103d0984d86f5"
format_stamp: "Formatted at 2026-09-26 02:06:48 on dist-test-slave-03rn"
server_key: "2a15cf5f6b8311b82504ab21da931e02"
server_key_iv: "0d90004ce940ac3b6aa7f64e7bb75afd"
server_key_version: "encryptionkey@0"
I20260926 02:06:48.048933 19614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "f2690709bf574d779fc103d0984d86f5"
format_stamp: "Formatted at 2026-09-26 02:06:48 on dist-test-slave-03rn"
server_key: "2a15cf5f6b8311b82504ab21da931e02"
server_key_iv: "0d90004ce940ac3b6aa7f64e7bb75afd"
server_key_version: "encryptionkey@0"
I20260926 02:06:48.054351 19614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:06:48.058061 19629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:48.059494 19614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:06:48.059682 19614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "f2690709bf574d779fc103d0984d86f5"
format_stamp: "Formatted at 2026-09-26 02:06:48 on dist-test-slave-03rn"
server_key: "2a15cf5f6b8311b82504ab21da931e02"
server_key_iv: "0d90004ce940ac3b6aa7f64e7bb75afd"
server_key_version: "encryptionkey@0"
I20260926 02:06:48.059893 19614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:06:48.074101 19614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:06:48.078527 19614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:06:48.078860 19614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:06:48.098296 19614 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:36911
I20260926 02:06:48.098315 19680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:36911 every 8 connection(s)
I20260926 02:06:48.099807 19614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:06:48.103866 19681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:06:48.107265 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 19614
I20260926 02:06:48.107527 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:06:48.108435 19070 external_mini_cluster.cc:1428] Setting key 003fe57541a93b920f2e810bf0b93428
I20260926 02:06:48.118952 19681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5: Bootstrap starting.
I20260926 02:06:48.123620 19681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5: Neither blocks nor log segments found. Creating new log.
Sep 26 02:06:48 dist-test-slave-03rn krb5kdc[19601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:06:48.125134 19681 log.cc:824] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5: Log is configured to *not* fsync() on all Append() calls
I20260926 02:06:48.129241 19681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5: No bootstrap required, opened a new log
I20260926 02:06:48.135210 19684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:48.110781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47220 (local address 127.18.159.190:36911)
0926 02:06:48.111603 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:48.111616 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:48.111667 (+ 51us) server_negotiation.cc:407] Connection header received
0926 02:06:48.113437 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:48.113462 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:48.113888 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:48.114268 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:48.116341 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:48.117727 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:48.119706 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:48.120530 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:48.120718 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:48.126013 (+ 5295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:48.126029 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:48.126048 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:48.126098 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:48.130372 (+ 4274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:48.131200 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:48.131211 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:48.131222 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:48.131460 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:48.131863 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:48.131868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:48.131872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:48.132711 (+ 839us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:48.133051 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:48.133609 (+ 558us) server_negotiation.cc:299] Negotiation successful
0926 02:06:48.134056 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":384,"threads_started":1}
I20260926 02:06:48.139153 19681 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2690709bf574d779fc103d0984d86f5" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36911 } }
I20260926 02:06:48.139611 19681 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:06:48.139760 19681 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2690709bf574d779fc103d0984d86f5, State: Initialized, Role: FOLLOWER
I20260926 02:06:48.140437 19681 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2690709bf574d779fc103d0984d86f5" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36911 } }
I20260926 02:06:48.140708 19681 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:06:48.140861 19681 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:06:48.141052 19681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:06:48.143476 19681 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2690709bf574d779fc103d0984d86f5" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36911 } }
I20260926 02:06:48.144062 19681 leader_election.cc:302] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2690709bf574d779fc103d0984d86f5; no voters:
I20260926 02:06:48.144835 19681 leader_election.cc:288] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:06:48.144940 19686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:06:48.145716 19686 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5 [term 1 LEADER]: Becoming Leader. State: Replica: f2690709bf574d779fc103d0984d86f5, State: Running, Role: LEADER
I20260926 02:06:48.146353 19686 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2690709bf574d779fc103d0984d86f5" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36911 } }
I20260926 02:06:48.147217 19681 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:06:48.149231 19688 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2690709bf574d779fc103d0984d86f5. Latest consensus state: current_term: 1 leader_uuid: "f2690709bf574d779fc103d0984d86f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2690709bf574d779fc103d0984d86f5" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36911 } } }
I20260926 02:06:48.150625 19688 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5 [sys.catalog]: This master's current role is: LEADER
I20260926 02:06:48.150317 19687 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2690709bf574d779fc103d0984d86f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2690709bf574d779fc103d0984d86f5" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36911 } } }
I20260926 02:06:48.151853 19687 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5 [sys.catalog]: This master's current role is: LEADER
I20260926 02:06:48.158655 19694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:06:48.163448 19694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:06:48.173063 19694 catalog_manager.cc:1347] Generated new cluster ID: 65eaec3d8a9e4a888e2d43193010b64f
I20260926 02:06:48.173202 19694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:06:48.186028 19694 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:06:48.187443 19694 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:06:48.196955 19694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2690709bf574d779fc103d0984d86f5: Generated new TSK 0
I20260926 02:06:48.197860 19694 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.159.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
I20260926 02:06:48.259562 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:36911
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:39597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:06:48.644325 19707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:06:48.644695 19707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:06:48.644800 19707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:06:48.652879 19707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:06:48.653276 19707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:06:48.653496 19707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:06:48.664920 19707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:39597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:36911
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:06:48.666365 19707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:06:48.667774 19707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:06:48.682104 19712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:48.682149 19713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:48.682137 19715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:06:48.683115 19707 server_base.cc:1034] running on GCE node
I20260926 02:06:48.683732 19707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:06:48.684772 19707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:06:48.685977 19707 hybrid_clock.cc:648] HybridClock initialized: now 1790388408685933 us; error 36 us; skew 500 ppm
I20260926 02:06:48.686389 19707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:06:48 dist-test-slave-03rn krb5kdc[19601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388408, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:06:48.689924 19707 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:06:48.691561 19707 webserver.cc:460] Webserver started at http://127.18.159.129:45417/ using document root <none> and password file <none>
I20260926 02:06:48.692353 19707 fs_manager.cc:359] Metadata directory not provided
I20260926 02:06:48.692467 19707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:06:48.692732 19707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:06:48.696033 19707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "8dccce0bc77140078e40dc86dcbe3473"
format_stamp: "Formatted at 2026-09-26 02:06:48 on dist-test-slave-03rn"
server_key: "e338676dbddc83e175bb5b0fcb566d9c"
server_key_iv: "97d04db9f49641bef1d22a3331e231f3"
server_key_version: "encryptionkey@0"
I20260926 02:06:48.697064 19707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "8dccce0bc77140078e40dc86dcbe3473"
format_stamp: "Formatted at 2026-09-26 02:06:48 on dist-test-slave-03rn"
server_key: "e338676dbddc83e175bb5b0fcb566d9c"
server_key_iv: "97d04db9f49641bef1d22a3331e231f3"
server_key_version: "encryptionkey@0"
I20260926 02:06:48.703591 19707 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 02:06:48.707722 19722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:48.709915 19707 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 02:06:48.710161 19707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "8dccce0bc77140078e40dc86dcbe3473"
format_stamp: "Formatted at 2026-09-26 02:06:48 on dist-test-slave-03rn"
server_key: "e338676dbddc83e175bb5b0fcb566d9c"
server_key_iv: "97d04db9f49641bef1d22a3331e231f3"
server_key_version: "encryptionkey@0"
I20260926 02:06:48.710400 19707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:06:48.723618 19707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:06:48.728722 19707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:06:48.729051 19707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:06:48.730093 19707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:06:48.731487 19707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:06:48.731559 19707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:48.731726 19707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:06:48.731801 19707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:48.767558 19707 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:42251
I20260926 02:06:48.767614 19834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:42251 every 8 connection(s)
I20260926 02:06:48.768929 19707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:06:48.771708 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 19707
I20260926 02:06:48.771938 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:06:48.772663 19070 external_mini_cluster.cc:1428] Setting key c9124d4797f6a9cb5f917125e17c47b6
Sep 26 02:06:48 dist-test-slave-03rn krb5kdc[19601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388408, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.159.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.130@KRBTEST.COM" created.
I20260926 02:06:48.793687 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:48.771714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:33351 (local address 127.18.159.190:36911)
0926 02:06:48.772278 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:48.772284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:48.773439 (+ 1155us) server_negotiation.cc:407] Connection header received
0926 02:06:48.775798 (+ 2359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:48.775807 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:48.775904 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:48.776029 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:48.777903 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:48.779091 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:48.782215 (+ 3124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:48.782643 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:48.782783 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:48.787607 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:48.787613 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:48.787616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:48.787654 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:48.790291 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:48.791140 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:48.791144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:48.791147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:48.791263 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:48.791730 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:48.791735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:48.791738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:48.792010 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:48.792171 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:48.793063 (+ 892us) server_negotiation.cc:299] Negotiation successful
0926 02:06:48.793301 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":252,"threads_started":1}
I20260926 02:06:48.794721 19838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:48.772343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:36911 (local address 127.18.159.129:33351)
0926 02:06:48.773217 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:48.773276 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:48.775454 (+ 2178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:48.776439 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:48.776448 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:48.777047 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:06:48.777691 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:48.777712 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:48.779302 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:48.779308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:48.781841 (+ 2533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:48.781872 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:48.782946 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:48.782952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:48.783331 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:48.784256 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:06:48.784292 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:06:48.787286 (+ 2994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:06:48.790478 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:48.790489 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:48.790509 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:48.790976 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:48.791387 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:48.791392 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:48.791396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:48.791603 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:48.792146 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:06:48.792182 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:06:48.792667 (+ 485us) client_negotiation.cc:769] Sending connection context
0926 02:06:48.792955 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 02:06:48.793572 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":339,"threads_started":1}
I20260926 02:06:48.796746 19835 heartbeater.cc:347] Connected to a master server at 127.18.159.190:36911
I20260926 02:06:48.797159 19835 heartbeater.cc:464] Registering TS with master...
I20260926 02:06:48.797927 19835 heartbeater.cc:511] Master 127.18.159.190:36911 requested a full tablet report, sending...
I20260926 02:06:48.800457 19646 ts_manager.cc:194] Registered new tserver with Master: 8dccce0bc77140078e40dc86dcbe3473 (127.18.159.129:42251)
I20260926 02:06:48.801967 19646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:33351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
I20260926 02:06:48.817724 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:36911
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:39597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:06:49.153107 19841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:06:49.153417 19841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:06:49.153602 19841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:06:49.161005 19841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:06:49.161125 19841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:06:49.161324 19841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:06:49.171232 19841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:39597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:36911
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:06:49.172688 19841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:06:49.174006 19841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:06:49.187492 19846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:49.187539 19849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:49.187561 19847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:06:49.188452 19841 server_base.cc:1034] running on GCE node
I20260926 02:06:49.189059 19841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:06:49.189987 19841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:06:49.191210 19841 hybrid_clock.cc:648] HybridClock initialized: now 1790388409191154 us; error 53 us; skew 500 ppm
I20260926 02:06:49.191599 19841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:06:49 dist-test-slave-03rn krb5kdc[19601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388409, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:06:49.195013 19841 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:06:49.196789 19841 webserver.cc:460] Webserver started at http://127.18.159.130:35381/ using document root <none> and password file <none>
I20260926 02:06:49.197661 19841 fs_manager.cc:359] Metadata directory not provided
I20260926 02:06:49.197770 19841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:06:49.198042 19841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:06:49.201264 19841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "caebb40de91a4fa080349b0570756ebe"
format_stamp: "Formatted at 2026-09-26 02:06:49 on dist-test-slave-03rn"
server_key: "458d77612d2b28f225294be31ac09427"
server_key_iv: "4d02b0f8b200ce93e440de7e0402284e"
server_key_version: "encryptionkey@0"
I20260926 02:06:49.202045 19841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "caebb40de91a4fa080349b0570756ebe"
format_stamp: "Formatted at 2026-09-26 02:06:49 on dist-test-slave-03rn"
server_key: "458d77612d2b28f225294be31ac09427"
server_key_iv: "4d02b0f8b200ce93e440de7e0402284e"
server_key_version: "encryptionkey@0"
I20260926 02:06:49.207614 19841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:06:49.211510 19856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:49.213172 19841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:06:49.213374 19841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "caebb40de91a4fa080349b0570756ebe"
format_stamp: "Formatted at 2026-09-26 02:06:49 on dist-test-slave-03rn"
server_key: "458d77612d2b28f225294be31ac09427"
server_key_iv: "4d02b0f8b200ce93e440de7e0402284e"
server_key_version: "encryptionkey@0"
I20260926 02:06:49.213572 19841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:06:49.237856 19841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:06:49.242620 19841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:06:49.242897 19841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:06:49.243916 19841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:06:49.245430 19841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:06:49.245512 19841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:49.245595 19841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:06:49.245638 19841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:49.280535 19841 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:45651
I20260926 02:06:49.280669 19968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:45651 every 8 connection(s)
I20260926 02:06:49.281919 19841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:06:49.286861 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 19841
I20260926 02:06:49.287092 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:06:49.287595 19070 external_mini_cluster.cc:1428] Setting key 6fa75d4b070102d80f0361c930eabe0d
Sep 26 02:06:49 dist-test-slave-03rn krb5kdc[19601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388409, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.159.131@KRBTEST.COM; defaulting to no policy
I20260926 02:06:49.304286 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:49.284786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:52041 (local address 127.18.159.190:36911)
0926 02:06:49.284991 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:49.284996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:49.286179 (+ 1183us) server_negotiation.cc:407] Connection header received
0926 02:06:49.288275 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:49.288281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:49.288387 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:49.288507 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:49.290179 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.290916 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.292765 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.293213 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.293363 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:49.298000 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:49.298005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:49.298008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:49.298046 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:49.301070 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:49.301899 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:49.301911 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:49.301915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:49.302043 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:49.302503 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:49.302507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:49.302509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:49.302774 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:49.302928 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:49.303661 (+ 733us) server_negotiation.cc:299] Negotiation successful
0926 02:06:49.303886 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:06:49.305481 19971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:49.285200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:36911 (local address 127.18.159.130:52041)
0926 02:06:49.285958 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:49.286004 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:49.287937 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:49.288784 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:49.288794 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:49.289332 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:06:49.289971 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:49.289995 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.291099 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.291105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:49.292598 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:49.292614 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.293472 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.293479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:49.293742 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:49.294829 (+ 1087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:06:49.294869 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:06:49.297699 (+ 2830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:06:49.301254 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:49.301263 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:49.301284 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:49.301754 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:49.302161 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:49.302167 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:49.302171 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:49.302371 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:49.302906 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:06:49.302914 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:06:49.303288 (+ 374us) client_negotiation.cc:769] Sending connection context
0926 02:06:49.303558 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 02:06:49.304182 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":318,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.131@KRBTEST.COM" created.
I20260926 02:06:49.307255 19969 heartbeater.cc:347] Connected to a master server at 127.18.159.190:36911
I20260926 02:06:49.307585 19969 heartbeater.cc:464] Registering TS with master...
I20260926 02:06:49.308395 19969 heartbeater.cc:511] Master 127.18.159.190:36911 requested a full tablet report, sending...
I20260926 02:06:49.310015 19646 ts_manager.cc:194] Registered new tserver with Master: caebb40de91a4fa080349b0570756ebe (127.18.159.130:45651)
I20260926 02:06:49.311208 19646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:52041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
I20260926 02:06:49.332011 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:36911
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:39597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:06:49.672206 19974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:06:49.672518 19974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:06:49.672618 19974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:06:49.680055 19974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:06:49.680271 19974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:06:49.680408 19974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:06:49.690697 19974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:39597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:36911
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:06:49.691999 19974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:06:49.693506 19974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:06:49.708246 19980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:49.708285 19982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:49.708211 19979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:06:49.709303 19974 server_base.cc:1034] running on GCE node
I20260926 02:06:49.709998 19974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:06:49.711027 19974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:06:49.712273 19974 hybrid_clock.cc:648] HybridClock initialized: now 1790388409712228 us; error 95 us; skew 500 ppm
I20260926 02:06:49.712661 19974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:06:49 dist-test-slave-03rn krb5kdc[19601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388409, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:06:49.716230 19974 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:06:49.717927 19974 webserver.cc:460] Webserver started at http://127.18.159.131:40915/ using document root <none> and password file <none>
I20260926 02:06:49.718770 19974 fs_manager.cc:359] Metadata directory not provided
I20260926 02:06:49.718904 19974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:06:49.719249 19974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:06:49.722465 19974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "15d1661517404328b434bd1da553a413"
format_stamp: "Formatted at 2026-09-26 02:06:49 on dist-test-slave-03rn"
server_key: "43917b76ea5e1615c359af20616c766b"
server_key_iv: "930238d25c4ddb5d3835bd02077efc96"
server_key_version: "encryptionkey@0"
I20260926 02:06:49.723381 19974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "15d1661517404328b434bd1da553a413"
format_stamp: "Formatted at 2026-09-26 02:06:49 on dist-test-slave-03rn"
server_key: "43917b76ea5e1615c359af20616c766b"
server_key_iv: "930238d25c4ddb5d3835bd02077efc96"
server_key_version: "encryptionkey@0"
I20260926 02:06:49.729342 19974 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 02:06:49.733263 19989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:49.734939 19974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:06:49.735168 19974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "15d1661517404328b434bd1da553a413"
format_stamp: "Formatted at 2026-09-26 02:06:49 on dist-test-slave-03rn"
server_key: "43917b76ea5e1615c359af20616c766b"
server_key_iv: "930238d25c4ddb5d3835bd02077efc96"
server_key_version: "encryptionkey@0"
I20260926 02:06:49.735386 19974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:06:49.754122 19974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:06:49.758708 19974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:06:49.759042 19974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:06:49.760196 19974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:06:49.761663 19974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:06:49.761744 19974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:49.761873 19974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:06:49.761926 19974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:49.795960 19974 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:35039
I20260926 02:06:49.796084 20101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:35039 every 8 connection(s)
I20260926 02:06:49.797443 19974 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:06:49.800976 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 19974
I20260926 02:06:49.801201 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:06:49.801889 19070 external_mini_cluster.cc:1428] Setting key 69bb515cc0743c3fe973850a4b465c41
I20260926 02:06:49.805485 19835 heartbeater.cc:503] Master 127.18.159.190:36911 was elected leader, sending a full tablet report...
Sep 26 02:06:49 dist-test-slave-03rn krb5kdc[19601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388409, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:06:49.821408 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:49.800041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:49813 (local address 127.18.159.190:36911)
0926 02:06:49.800313 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:49.800319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:49.801877 (+ 1558us) server_negotiation.cc:407] Connection header received
0926 02:06:49.805323 (+ 3446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:49.805329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:49.805419 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:49.805542 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:49.807797 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.808786 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.810966 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.811437 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.811552 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:49.815315 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:49.815320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:49.815323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:49.815358 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:49.818327 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:49.819354 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:49.819359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:49.819363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:49.819492 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:49.819885 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:49.819888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:49.819890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:49.820143 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:49.820322 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:49.821000 (+ 678us) server_negotiation.cc:299] Negotiation successful
0926 02:06:49.821171 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260926 02:06:49.822501 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:49.800631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:36911 (local address 127.18.159.131:49813)
0926 02:06:49.801652 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:49.801721 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:49.804935 (+ 3214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:49.805868 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:49.805884 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:49.806758 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:06:49.807630 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:49.807650 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.809049 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.809057 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:49.810810 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:49.810824 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.811649 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.811654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:49.811872 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:49.812642 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:06:49.812666 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:06:49.815064 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:06:49.818495 (+ 3431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:49.818503 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:49.818519 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:49.819227 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:49.819589 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:49.819592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:49.819594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:49.819791 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:49.820303 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:06:49.820309 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:06:49.820670 (+ 361us) client_negotiation.cc:769] Sending connection context
0926 02:06:49.820907 (+ 237us) client_negotiation.cc:240] Negotiation successful
0926 02:06:49.821319 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":358,"threads_started":1}
I20260926 02:06:49.824322 20102 heartbeater.cc:347] Connected to a master server at 127.18.159.190:36911
I20260926 02:06:49.824669 20102 heartbeater.cc:464] Registering TS with master...
I20260926 02:06:49.825475 20102 heartbeater.cc:511] Master 127.18.159.190:36911 requested a full tablet report, sending...
I20260926 02:06:49.827128 19646 ts_manager.cc:194] Registered new tserver with Master: 15d1661517404328b434bd1da553a413 (127.18.159.131:35039)
I20260926 02:06:49.827983 19646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:49813
I20260926 02:06:49.831864 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:06:49.851504 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:49.838687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.18.159.190:36911)
0926 02:06:49.838886 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:49.838892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:49.839368 (+ 476us) server_negotiation.cc:407] Connection header received
0926 02:06:49.839562 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:49.839570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:49.839637 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:49.839740 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:49.841228 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.841940 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.843202 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.843595 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.843723 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:49.846023 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:49.846027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:49.846031 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:49.846069 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:49.848813 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:49.849479 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:49.849486 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:49.849490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:49.849632 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:49.850005 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:49.850012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:49.850015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:49.850338 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:49.850615 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:49.850969 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 02:06:49.851192 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 02:06:49.858516 19646 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:06:49.866101 19646 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:06:49.909909 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:49.878892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:42251 (local address 127.0.0.1:55292)
0926 02:06:49.897164 (+ 18272us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:49.897195 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:49.897497 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:49.898790 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:49.898801 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:49.898846 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:06:49.899387 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:49.899403 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.902159 (+ 2756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.902167 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:49.905590 (+ 3423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:49.905615 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.908868 (+ 3253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.908877 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:49.909109 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:49.909167 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 02:06:49.909306 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 02:06:49.909500 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18102,"mutex_wait_us":30,"thread_start_us":23683,"threads_started":1}
I20260926 02:06:49.910919 20114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:49.878421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:35039 (local address 127.0.0.1:47046)
0926 02:06:49.895464 (+ 17043us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:49.895508 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:49.895688 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:49.899503 (+ 3815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:49.899511 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:49.899546 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:06:49.900031 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:49.900046 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.902603 (+ 2557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.902611 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:49.907435 (+ 4824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:49.907459 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.910084 (+ 2625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.910091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:49.910281 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:49.910309 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 02:06:49.910379 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 02:06:49.910517 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16798,"mutex_wait_us":3,"thread_start_us":16647,"threads_started":1}
I20260926 02:06:49.911033 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:49.878787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47046 (local address 127.18.159.131:35039)
0926 02:06:49.898370 (+ 19583us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:49.898383 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:49.898427 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:06:49.898552 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:49.898564 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:49.898873 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:49.899157 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:06:49.900346 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.902389 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.907708 (+ 5319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.909863 (+ 2155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.910090 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:49.910266 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:49.910515 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 02:06:49.910697 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17816,"thread_start_us":17659,"threads_started":1}
I20260926 02:06:49.911610 20117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:49.879562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55292 (local address 127.18.159.129:42251)
0926 02:06:49.894639 (+ 15077us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:49.894654 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:49.897346 (+ 2692us) server_negotiation.cc:407] Connection header received
0926 02:06:49.897770 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:49.897783 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:49.898146 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:49.898370 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:06:49.899641 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.901879 (+ 2238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.905854 (+ 3975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.908062 (+ 2208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.908369 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:49.908647 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:49.910436 (+ 1789us) server_negotiation.cc:299] Negotiation successful
0926 02:06:49.910679 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14807,"thread_start_us":15095,"threads_started":1}
I20260926 02:06:49.916446 19770 tablet_service.cc:1467] Processing CreateTablet for tablet c1c9b9446b054c4f8539540d2d292f89 (DEFAULT_TABLE table=test-table [id=a389fbd56d054fb3816c4070aa5764bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:06:49.917105 20037 tablet_service.cc:1467] Processing CreateTablet for tablet c1c9b9446b054c4f8539540d2d292f89 (DEFAULT_TABLE table=test-table [id=a389fbd56d054fb3816c4070aa5764bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:06:49.917922 19770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c9b9446b054c4f8539540d2d292f89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:06:49.918551 20037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c9b9446b054c4f8539540d2d292f89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:06:49.919061 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:49.896632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:45651 (local address 127.0.0.1:42622)
0926 02:06:49.910182 (+ 13550us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:49.910203 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:49.910315 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:49.911972 (+ 1657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:49.911976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:49.912004 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:06:49.912646 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:49.912665 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.914545 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.914561 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:49.916327 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:49.916341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.918196 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.918202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:49.918374 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:49.918406 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 02:06:49.918497 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 02:06:49.918751 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13450,"thread_start_us":17005,"threads_started":1}
I20260926 02:06:49.919086 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:49.896660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42622 (local address 127.18.159.130:45651)
0926 02:06:49.909263 (+ 12603us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:49.909275 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:49.911308 (+ 2033us) server_negotiation.cc:407] Connection header received
0926 02:06:49.911421 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:49.911429 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:49.911747 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:49.912803 (+ 1056us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:06:49.912985 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.914372 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.916506 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:49.917939 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:49.918204 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:49.918376 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:49.918587 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 02:06:49.918750 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12428,"thread_start_us":12612,"threads_started":1}
I20260926 02:06:49.922629 19904 tablet_service.cc:1467] Processing CreateTablet for tablet c1c9b9446b054c4f8539540d2d292f89 (DEFAULT_TABLE table=test-table [id=a389fbd56d054fb3816c4070aa5764bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:06:49.923789 19904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c9b9446b054c4f8539540d2d292f89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:06:49.934325 20120 tablet_bootstrap.cc:492] T c1c9b9446b054c4f8539540d2d292f89 P 15d1661517404328b434bd1da553a413: Bootstrap starting.
I20260926 02:06:49.939595 20120 tablet_bootstrap.cc:654] T c1c9b9446b054c4f8539540d2d292f89 P 15d1661517404328b434bd1da553a413: Neither blocks nor log segments found. Creating new log.
I20260926 02:06:49.940038 20121 tablet_bootstrap.cc:492] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473: Bootstrap starting.
I20260926 02:06:49.940658 20122 tablet_bootstrap.cc:492] T c1c9b9446b054c4f8539540d2d292f89 P caebb40de91a4fa080349b0570756ebe: Bootstrap starting.
I20260926 02:06:49.941754 20120 log.cc:824] T c1c9b9446b054c4f8539540d2d292f89 P 15d1661517404328b434bd1da553a413: Log is configured to *not* fsync() on all Append() calls
I20260926 02:06:49.944600 20121 tablet_bootstrap.cc:654] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473: Neither blocks nor log segments found. Creating new log.
I20260926 02:06:49.944641 20122 tablet_bootstrap.cc:654] T c1c9b9446b054c4f8539540d2d292f89 P caebb40de91a4fa080349b0570756ebe: Neither blocks nor log segments found. Creating new log.
I20260926 02:06:49.945852 20121 log.cc:824] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473: Log is configured to *not* fsync() on all Append() calls
I20260926 02:06:49.945859 20122 log.cc:824] T c1c9b9446b054c4f8539540d2d292f89 P caebb40de91a4fa080349b0570756ebe: Log is configured to *not* fsync() on all Append() calls
I20260926 02:06:49.946398 20120 tablet_bootstrap.cc:492] T c1c9b9446b054c4f8539540d2d292f89 P 15d1661517404328b434bd1da553a413: No bootstrap required, opened a new log
I20260926 02:06:49.946779 20120 ts_tablet_manager.cc:1397] T c1c9b9446b054c4f8539540d2d292f89 P 15d1661517404328b434bd1da553a413: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20260926 02:06:49.949173 20122 tablet_bootstrap.cc:492] T c1c9b9446b054c4f8539540d2d292f89 P caebb40de91a4fa080349b0570756ebe: No bootstrap required, opened a new log
I20260926 02:06:49.949357 20121 tablet_bootstrap.cc:492] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473: No bootstrap required, opened a new log
I20260926 02:06:49.949514 20122 ts_tablet_manager.cc:1397] T c1c9b9446b054c4f8539540d2d292f89 P caebb40de91a4fa080349b0570756ebe: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260926 02:06:49.949635 20121 ts_tablet_manager.cc:1397] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 02:06:49.956947 20122 raft_consensus.cc:348] T c1c9b9446b054c4f8539540d2d292f89 P caebb40de91a4fa080349b0570756ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d1661517404328b434bd1da553a413" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35039 } } peers { permanent_uuid: "8dccce0bc77140078e40dc86dcbe3473" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42251 } } peers { permanent_uuid: "caebb40de91a4fa080349b0570756ebe" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45651 } }
I20260926 02:06:49.957413 20122 raft_consensus.cc:374] T c1c9b9446b054c4f8539540d2d292f89 P caebb40de91a4fa080349b0570756ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:06:49.957511 20122 raft_consensus.cc:729] T c1c9b9446b054c4f8539540d2d292f89 P caebb40de91a4fa080349b0570756ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caebb40de91a4fa080349b0570756ebe, State: Initialized, Role: FOLLOWER
I20260926 02:06:49.958388 20120 raft_consensus.cc:348] T c1c9b9446b054c4f8539540d2d292f89 P 15d1661517404328b434bd1da553a413 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d1661517404328b434bd1da553a413" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35039 } } peers { permanent_uuid: "8dccce0bc77140078e40dc86dcbe3473" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42251 } } peers { permanent_uuid: "caebb40de91a4fa080349b0570756ebe" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45651 } }
I20260926 02:06:49.958667 20122 consensus_queue.cc:260] T c1c9b9446b054c4f8539540d2d292f89 P caebb40de91a4fa080349b0570756ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d1661517404328b434bd1da553a413" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35039 } } peers { permanent_uuid: "8dccce0bc77140078e40dc86dcbe3473" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42251 } } peers { permanent_uuid: "caebb40de91a4fa080349b0570756ebe" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45651 } }
I20260926 02:06:49.958890 20120 raft_consensus.cc:374] T c1c9b9446b054c4f8539540d2d292f89 P 15d1661517404328b434bd1da553a413 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:06:49.959040 20120 raft_consensus.cc:729] T c1c9b9446b054c4f8539540d2d292f89 P 15d1661517404328b434bd1da553a413 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d1661517404328b434bd1da553a413, State: Initialized, Role: FOLLOWER
I20260926 02:06:49.959738 20120 consensus_queue.cc:260] T c1c9b9446b054c4f8539540d2d292f89 P 15d1661517404328b434bd1da553a413 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d1661517404328b434bd1da553a413" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35039 } } peers { permanent_uuid: "8dccce0bc77140078e40dc86dcbe3473" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42251 } } peers { permanent_uuid: "caebb40de91a4fa080349b0570756ebe" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45651 } }
I20260926 02:06:49.960400 19969 heartbeater.cc:503] Master 127.18.159.190:36911 was elected leader, sending a full tablet report...
I20260926 02:06:49.961226 20121 raft_consensus.cc:348] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d1661517404328b434bd1da553a413" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35039 } } peers { permanent_uuid: "8dccce0bc77140078e40dc86dcbe3473" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42251 } } peers { permanent_uuid: "caebb40de91a4fa080349b0570756ebe" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45651 } }
I20260926 02:06:49.961687 20121 raft_consensus.cc:374] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:06:49.961796 20121 raft_consensus.cc:729] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dccce0bc77140078e40dc86dcbe3473, State: Initialized, Role: FOLLOWER
I20260926 02:06:49.961884 20122 ts_tablet_manager.cc:1428] T c1c9b9446b054c4f8539540d2d292f89 P caebb40de91a4fa080349b0570756ebe: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260926 02:06:49.962988 20102 heartbeater.cc:503] Master 127.18.159.190:36911 was elected leader, sending a full tablet report...
I20260926 02:06:49.963938 20120 ts_tablet_manager.cc:1428] T c1c9b9446b054c4f8539540d2d292f89 P 15d1661517404328b434bd1da553a413: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260926 02:06:49.963819 20121 consensus_queue.cc:260] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d1661517404328b434bd1da553a413" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35039 } } peers { permanent_uuid: "8dccce0bc77140078e40dc86dcbe3473" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42251 } } peers { permanent_uuid: "caebb40de91a4fa080349b0570756ebe" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45651 } }
I20260926 02:06:49.965850 20121 ts_tablet_manager.cc:1428] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260926 02:06:50.021266 20128 raft_consensus.cc:482] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260926 02:06:50.021754 19836 tablet.cc:2369] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:06:50.021816 20128 raft_consensus.cc:504] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d1661517404328b434bd1da553a413" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35039 } } peers { permanent_uuid: "8dccce0bc77140078e40dc86dcbe3473" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42251 } } peers { permanent_uuid: "caebb40de91a4fa080349b0570756ebe" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45651 } }
I20260926 02:06:50.023640 20128 leader_election.cc:288] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15d1661517404328b434bd1da553a413 (127.18.159.131:35039), caebb40de91a4fa080349b0570756ebe (127.18.159.130:45651)
I20260926 02:06:50.031567 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:50.024062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:55653 (local address 127.18.159.131:35039)
0926 02:06:50.024299 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:50.024304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:50.024840 (+ 536us) server_negotiation.cc:407] Connection header received
0926 02:06:50.025139 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:50.025148 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:50.025256 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:50.025342 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:06:50.026047 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:50.027644 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:50.029740 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:50.030518 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:50.030729 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:50.030813 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:50.031159 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 02:06:50.031288 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:06:50.031579 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:50.025124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:58045 (local address 127.18.159.130:45651)
0926 02:06:50.025348 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:50.025355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:50.025642 (+ 287us) server_negotiation.cc:407] Connection header received
0926 02:06:50.025879 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:50.025884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:50.025975 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:50.026067 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:06:50.026835 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:50.027821 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:50.029751 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:50.030544 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:50.030735 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:50.030822 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:50.031172 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 02:06:50.031305 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:06:50.031733 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:50.024846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:45651 (local address 127.18.159.129:58045)
0926 02:06:50.025525 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:50.025539 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:50.025734 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:50.026082 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:50.026085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:50.026102 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:06:50.026668 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:50.026679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:50.027968 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:50.027974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:50.029556 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:50.029573 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:50.030807 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:50.030813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:50.030994 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:50.031033 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 02:06:50.031144 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 02:06:50.031283 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":47,"thread_start_us":338,"threads_started":1}
I20260926 02:06:50.031802 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:50.023979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:35039 (local address 127.18.159.129:55653)
0926 02:06:50.024685 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:50.024707 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:50.024857 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:50.025518 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:50.025522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:50.025543 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:06:50.025897 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:50.025905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:50.027790 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:50.027797 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:50.029553 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:50.029568 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:50.030776 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:50.030787 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:50.030957 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:50.031013 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 02:06:50.031120 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 02:06:50.031284 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":76,"spinlock_wait_cycles":65920,"thread_start_us":402,"threads_started":1}
I20260926 02:06:50.033146 19924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c9b9446b054c4f8539540d2d292f89" candidate_uuid: "8dccce0bc77140078e40dc86dcbe3473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caebb40de91a4fa080349b0570756ebe" is_pre_election: true
I20260926 02:06:50.033120 20057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c9b9446b054c4f8539540d2d292f89" candidate_uuid: "8dccce0bc77140078e40dc86dcbe3473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d1661517404328b434bd1da553a413" is_pre_election: true
I20260926 02:06:50.033758 19924 raft_consensus.cc:2454] T c1c9b9446b054c4f8539540d2d292f89 P caebb40de91a4fa080349b0570756ebe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dccce0bc77140078e40dc86dcbe3473 in term 0.
I20260926 02:06:50.033758 20057 raft_consensus.cc:2454] T c1c9b9446b054c4f8539540d2d292f89 P 15d1661517404328b434bd1da553a413 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dccce0bc77140078e40dc86dcbe3473 in term 0.
W20260926 02:06:50.034099 19970 tablet.cc:2369] T c1c9b9446b054c4f8539540d2d292f89 P caebb40de91a4fa080349b0570756ebe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:06:50.034572 19723 leader_election.cc:302] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dccce0bc77140078e40dc86dcbe3473, caebb40de91a4fa080349b0570756ebe; no voters:
I20260926 02:06:50.035036 20128 raft_consensus.cc:2781] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:06:50.035207 20128 raft_consensus.cc:482] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:06:50.035317 20128 raft_consensus.cc:3037] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:06:50.037597 20128 raft_consensus.cc:504] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d1661517404328b434bd1da553a413" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35039 } } peers { permanent_uuid: "8dccce0bc77140078e40dc86dcbe3473" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42251 } } peers { permanent_uuid: "caebb40de91a4fa080349b0570756ebe" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45651 } }
I20260926 02:06:50.038736 20128 leader_election.cc:288] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473 [CANDIDATE]: Term 1 election: Requested vote from peers 15d1661517404328b434bd1da553a413 (127.18.159.131:35039), caebb40de91a4fa080349b0570756ebe (127.18.159.130:45651)
I20260926 02:06:50.039237 20057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c9b9446b054c4f8539540d2d292f89" candidate_uuid: "8dccce0bc77140078e40dc86dcbe3473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d1661517404328b434bd1da553a413"
I20260926 02:06:50.039263 19924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c9b9446b054c4f8539540d2d292f89" candidate_uuid: "8dccce0bc77140078e40dc86dcbe3473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caebb40de91a4fa080349b0570756ebe"
I20260926 02:06:50.039469 20057 raft_consensus.cc:3037] T c1c9b9446b054c4f8539540d2d292f89 P 15d1661517404328b434bd1da553a413 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:06:50.039479 19924 raft_consensus.cc:3037] T c1c9b9446b054c4f8539540d2d292f89 P caebb40de91a4fa080349b0570756ebe [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:06:50.041728 19924 raft_consensus.cc:2454] T c1c9b9446b054c4f8539540d2d292f89 P caebb40de91a4fa080349b0570756ebe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dccce0bc77140078e40dc86dcbe3473 in term 1.
I20260926 02:06:50.041963 20057 raft_consensus.cc:2454] T c1c9b9446b054c4f8539540d2d292f89 P 15d1661517404328b434bd1da553a413 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dccce0bc77140078e40dc86dcbe3473 in term 1.
I20260926 02:06:50.042223 19723 leader_election.cc:302] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dccce0bc77140078e40dc86dcbe3473, caebb40de91a4fa080349b0570756ebe; no voters:
I20260926 02:06:50.042518 20128 raft_consensus.cc:2781] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:06:50.043052 20128 raft_consensus.cc:686] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473 [term 1 LEADER]: Becoming Leader. State: Replica: 8dccce0bc77140078e40dc86dcbe3473, State: Running, Role: LEADER
I20260926 02:06:50.043550 20128 consensus_queue.cc:237] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d1661517404328b434bd1da553a413" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35039 } } peers { permanent_uuid: "8dccce0bc77140078e40dc86dcbe3473" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42251 } } peers { permanent_uuid: "caebb40de91a4fa080349b0570756ebe" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45651 } }
W20260926 02:06:50.049062 20103 tablet.cc:2369] T c1c9b9446b054c4f8539540d2d292f89 P 15d1661517404328b434bd1da553a413: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:06:50.049052 19644 catalog_manager.cc:5495] T c1c9b9446b054c4f8539540d2d292f89 P 8dccce0bc77140078e40dc86dcbe3473 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dccce0bc77140078e40dc86dcbe3473 (127.18.159.129). New cstate: current_term: 1 leader_uuid: "8dccce0bc77140078e40dc86dcbe3473" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d1661517404328b434bd1da553a413" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dccce0bc77140078e40dc86dcbe3473" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "caebb40de91a4fa080349b0570756ebe" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45651 } health_report { overall_health: UNKNOWN } } }
Sep 26 02:06:50 dist-test-slave-03rn krb5kdc[19601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:06:50.094017 20117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:50.076363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55296 (local address 127.18.159.129:42251)
0926 02:06:50.076656 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:50.076664 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:50.077071 (+ 407us) server_negotiation.cc:407] Connection header received
0926 02:06:50.077294 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:50.077298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:50.077363 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:50.077497 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:50.079554 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:50.080374 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:50.081710 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:50.082142 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:50.082287 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:50.087132 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:50.087150 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:50.087166 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:50.087209 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:50.090553 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:50.091538 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:50.091547 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:50.091549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:50.091741 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:50.092238 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:50.092242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:50.092244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:50.092704 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:50.093025 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:50.093368 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 02:06:50.093688 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
Sep 26 02:06:50 dist-test-slave-03rn krb5kdc[19601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388410, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 02:06:50 dist-test-slave-03rn krb5kdc[19601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388410, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:06:50.145784 20117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:50.130627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55306 (local address 127.18.159.129:42251)
0926 02:06:50.130914 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:50.130921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:50.131230 (+ 309us) server_negotiation.cc:407] Connection header received
0926 02:06:50.131459 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:50.131464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:50.131525 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:50.131658 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:50.133100 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:50.133912 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:50.135281 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:50.135701 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:50.135839 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:50.140291 (+ 4452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:50.140297 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:50.140300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:50.140340 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:50.142944 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:50.143659 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:50.143666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:50.143670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:50.143815 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:50.144307 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:50.144313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:50.144319 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:50.144618 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:50.144909 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:50.145199 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 02:06:50.145428 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
Sep 26 02:06:50 dist-test-slave-03rn krb5kdc[19601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:06:50 dist-test-slave-03rn krb5kdc[19601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:06:50.194022 20117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:50.178803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.18.159.129:42251)
0926 02:06:50.179164 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:50.179172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:50.179350 (+ 178us) server_negotiation.cc:407] Connection header received
0926 02:06:50.179502 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:50.179506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:50.179589 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:50.179681 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:50.181435 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:50.182264 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:50.183615 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:50.184089 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:50.184329 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:50.188693 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:50.188699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:50.188702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:50.188742 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:50.191401 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:50.192146 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:50.192150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:50.192152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:50.192353 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:50.192728 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:50.192731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:50.192733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:50.192996 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:50.193191 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:50.193515 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 02:06:50.193681 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
W20260926 02:06:50.195824 19750 tablet_service.cc:2908] Rejecting scan request for tablet c1c9b9446b054c4f8539540d2d292f89: Uninitialized: clean time has not yet been initialized
I20260926 02:06:50.199258 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 19707
I20260926 02:06:50.213248 20117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:50.200375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.18.159.129:42251)
0926 02:06:50.200549 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:50.200553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:50.200879 (+ 326us) server_negotiation.cc:407] Connection header received
0926 02:06:50.201029 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:50.201040 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:50.201136 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:50.201227 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:50.202526 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:50.203252 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:50.204606 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:50.205017 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:50.205154 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:50.207462 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:50.207467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:50.207471 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:50.207505 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:50.210625 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:50.211408 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:50.211412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:50.211414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:50.211552 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:50.211875 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:50.211878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:50.211880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:50.212269 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:50.212445 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:50.212759 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 02:06:50.212954 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:06:50.213985 19830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:06:50.307879 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 19707
I20260926 02:06:50.325076 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 19841
Sep 26 02:06:50 dist-test-slave-03rn krb5kdc[19601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.130@KRBTEST.COM
I20260926 02:06:50.339998 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:50.325995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42624 (local address 127.18.159.130:45651)
0926 02:06:50.326198 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:50.326204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:50.326225 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:06:50.326340 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:50.326344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:50.326410 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:50.326515 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:50.327662 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:50.328486 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:50.329746 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:50.330151 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:50.330280 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:50.334609 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:50.334627 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:50.334643 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:50.334683 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:50.337411 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:50.338139 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:50.338146 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:50.338148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:50.338272 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:50.338578 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:50.338581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:50.338584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:50.338943 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:50.339262 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:50.339435 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 02:06:50.339652 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:06:50.340826 19964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:06:50.432524 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 19841
I20260926 02:06:50.448837 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 19974
Sep 26 02:06:50 dist-test-slave-03rn krb5kdc[19601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.131@KRBTEST.COM
I20260926 02:06:50.466215 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:50.449680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47056 (local address 127.18.159.131:35039)
0926 02:06:50.449847 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:50.449851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:50.449872 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:06:50.449970 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:50.449974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:50.450037 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:50.450170 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:50.451494 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:50.452418 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:50.453962 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:50.454417 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:50.454547 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:50.459242 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:50.459259 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:50.459275 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:50.459322 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:50.462423 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:50.463431 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:50.463444 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:50.463448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:50.463642 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:50.464297 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:50.464303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:50.464306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:50.464979 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:50.465226 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:50.465613 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 02:06:50.465807 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:06:50.466954 20097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:06:50.555539 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 19974
I20260926 02:06:50.571561 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 19614
I20260926 02:06:50.572824 19676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:06:50.680442 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 19614
2026-09-26T02:06:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4313 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:06:50 dist-test-slave-03rn krb5kdc[20150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:06:50 dist-test-slave-03rn krb5kdc[20150](info): set up 2 sockets
Sep 26 02:06:50 dist-test-slave-03rn krb5kdc[20150](info): commencing operation
krb5kdc: starting...
W20260926 02:06:51.825140 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:06:51 dist-test-slave-03rn krb5kdc[20150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:06:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:06:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:06:52.016645 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:45847
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:38343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:45847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:06:52.382150 20163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:06:52.382493 20163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:06:52.382587 20163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:06:52.390719 20163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:06:52.390858 20163 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 02:06:52.390901 20163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:06:52.390956 20163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:06:52.391038 20163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:06:52.391095 20163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:06:52.401710 20163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:38343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:45847
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:45847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:06:52.403031 20163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:06:52.404453 20163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:06:52.417927 20169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:52.417951 20168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:06:52.418480 20163 server_base.cc:1034] running on GCE node
W20260926 02:06:52.417925 20171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:06:52.419651 20163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:06:52.421406 20163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:06:52.422693 20163 hybrid_clock.cc:648] HybridClock initialized: now 1790388412422608 us; error 71 us; skew 500 ppm
I20260926 02:06:52.423061 20163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:06:52 dist-test-slave-03rn krb5kdc[20150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388412, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:06:52.426594 20163 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:06:52.428349 20163 webserver.cc:460] Webserver started at http://127.18.159.190:33365/ using document root <none> and password file <none>
I20260926 02:06:52.429137 20163 fs_manager.cc:359] Metadata directory not provided
I20260926 02:06:52.429280 20163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:06:52.429632 20163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:06:52.433266 20163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "bab9c0d8559c4bbda907f8225a486133"
format_stamp: "Formatted at 2026-09-26 02:06:52 on dist-test-slave-03rn"
server_key: "45cc822a76ee7b5ba6f134a5da71a4b4"
server_key_iv: "af53baf1fecb3a75ee70c0ffc2d59f3b"
server_key_version: "encryptionkey@0"
I20260926 02:06:52.434135 20163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "bab9c0d8559c4bbda907f8225a486133"
format_stamp: "Formatted at 2026-09-26 02:06:52 on dist-test-slave-03rn"
server_key: "45cc822a76ee7b5ba6f134a5da71a4b4"
server_key_iv: "af53baf1fecb3a75ee70c0ffc2d59f3b"
server_key_version: "encryptionkey@0"
I20260926 02:06:52.439916 20163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 02:06:52.444756 20178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:52.446415 20163 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:06:52.446643 20163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "bab9c0d8559c4bbda907f8225a486133"
format_stamp: "Formatted at 2026-09-26 02:06:52 on dist-test-slave-03rn"
server_key: "45cc822a76ee7b5ba6f134a5da71a4b4"
server_key_iv: "af53baf1fecb3a75ee70c0ffc2d59f3b"
server_key_version: "encryptionkey@0"
I20260926 02:06:52.446906 20163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:06:52.466018 20163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:06:52.470779 20163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:06:52.471086 20163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:06:52.494032 20163 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:45847
I20260926 02:06:52.494055 20239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:45847 every 8 connection(s)
I20260926 02:06:52.495893 20163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:06:52.497126 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 20163
I20260926 02:06:52.497393 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:06:52.498024 19070 external_mini_cluster.cc:1428] Setting key 6fe6a8005cc451718cdb1e8ff05b8e9e
I20260926 02:06:52.500824 20240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:06:52 dist-test-slave-03rn krb5kdc[20150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:06:52.515209 20240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133: Bootstrap starting.
I20260926 02:06:52.519804 20240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133: Neither blocks nor log segments found. Creating new log.
I20260926 02:06:52.521212 20240 log.cc:824] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133: Log is configured to *not* fsync() on all Append() calls
I20260926 02:06:52.524008 20243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:52.500044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.18.159.190:45847)
0926 02:06:52.500876 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:52.500889 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:52.500930 (+ 41us) server_negotiation.cc:407] Connection header received
0926 02:06:52.502571 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:52.502594 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:52.503027 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:52.503399 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:52.505356 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:52.506820 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:52.508701 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:52.509431 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:52.509576 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:52.514616 (+ 5040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:52.514630 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:52.514657 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:52.514715 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:52.518836 (+ 4121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:52.519701 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:52.519712 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:52.519722 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:52.519966 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:52.520441 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:52.520446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:52.520449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:52.521176 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:52.521487 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:52.521966 (+ 479us) server_negotiation.cc:299] Negotiation successful
0926 02:06:52.522518 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":368,"threads_started":1}
I20260926 02:06:52.525228 20240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133: No bootstrap required, opened a new log
I20260926 02:06:52.534431 20240 raft_consensus.cc:348] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab9c0d8559c4bbda907f8225a486133" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45847 } }
I20260926 02:06:52.534873 20240 raft_consensus.cc:374] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:06:52.534957 20240 raft_consensus.cc:729] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bab9c0d8559c4bbda907f8225a486133, State: Initialized, Role: FOLLOWER
I20260926 02:06:52.535517 20240 consensus_queue.cc:260] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab9c0d8559c4bbda907f8225a486133" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45847 } }
I20260926 02:06:52.535723 20240 raft_consensus.cc:388] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:06:52.535828 20240 raft_consensus.cc:482] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:06:52.535969 20240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:06:52.538506 20240 raft_consensus.cc:504] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab9c0d8559c4bbda907f8225a486133" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45847 } }
I20260926 02:06:52.539222 20240 leader_election.cc:302] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bab9c0d8559c4bbda907f8225a486133; no voters:
I20260926 02:06:52.540037 20240 leader_election.cc:288] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:06:52.540186 20245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:06:52.540995 20245 raft_consensus.cc:686] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133 [term 1 LEADER]: Becoming Leader. State: Replica: bab9c0d8559c4bbda907f8225a486133, State: Running, Role: LEADER
I20260926 02:06:52.541595 20245 consensus_queue.cc:237] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab9c0d8559c4bbda907f8225a486133" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45847 } }
I20260926 02:06:52.542832 20240 sys_catalog.cc:564] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:06:52.545220 20247 sys_catalog.cc:455] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bab9c0d8559c4bbda907f8225a486133. Latest consensus state: current_term: 1 leader_uuid: "bab9c0d8559c4bbda907f8225a486133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab9c0d8559c4bbda907f8225a486133" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45847 } } }
I20260926 02:06:52.545337 20246 sys_catalog.cc:455] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bab9c0d8559c4bbda907f8225a486133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab9c0d8559c4bbda907f8225a486133" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45847 } } }
I20260926 02:06:52.545953 20246 sys_catalog.cc:458] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133 [sys.catalog]: This master's current role is: LEADER
I20260926 02:06:52.545961 20247 sys_catalog.cc:458] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133 [sys.catalog]: This master's current role is: LEADER
I20260926 02:06:52.547928 20250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:06:52.553983 20250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:06:52.563426 20250 catalog_manager.cc:1347] Generated new cluster ID: fe23119cdc61471f9e79af4b41f49019
I20260926 02:06:52.563580 20250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:06:52.593250 20250 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:06:52.595202 20250 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:06:52.612425 20250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133: Generated new TSK 0
I20260926 02:06:52.613524 20250 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.159.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
I20260926 02:06:52.660573 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:45847
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:38343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:06:53.012331 20266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:06:53.012660 20266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:06:53.012825 20266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:06:53.020889 20266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:06:53.021062 20266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:06:53.021205 20266 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:06:53.021338 20266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:06:53.031745 20266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:38343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.159.190:45847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:06:53.033187 20266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:06:53.034560 20266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:06:53.048012 20272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:53.048295 20271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:53.048671 20274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:06:53.049041 20266 server_base.cc:1034] running on GCE node
I20260926 02:06:53.049705 20266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:06:53.050743 20266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:06:53.051970 20266 hybrid_clock.cc:648] HybridClock initialized: now 1790388413051856 us; error 80 us; skew 500 ppm
I20260926 02:06:53.052395 20266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:06:53 dist-test-slave-03rn krb5kdc[20150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388413, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:06:53.061729 20266 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:06:53.063340 20266 webserver.cc:460] Webserver started at http://127.18.159.129:44931/ using document root <none> and password file <none>
I20260926 02:06:53.064110 20266 fs_manager.cc:359] Metadata directory not provided
I20260926 02:06:53.064297 20266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:06:53.064661 20266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:06:53.067693 20266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "132776399a2241e8b33be7693e69fd30"
format_stamp: "Formatted at 2026-09-26 02:06:53 on dist-test-slave-03rn"
server_key: "e8ea55f45f57b91756cb81d92d3cfbe9"
server_key_iv: "c36ccdbb1cb9d9a67b2ee993b72940d9"
server_key_version: "encryptionkey@0"
I20260926 02:06:53.068598 20266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "132776399a2241e8b33be7693e69fd30"
format_stamp: "Formatted at 2026-09-26 02:06:53 on dist-test-slave-03rn"
server_key: "e8ea55f45f57b91756cb81d92d3cfbe9"
server_key_iv: "c36ccdbb1cb9d9a67b2ee993b72940d9"
server_key_version: "encryptionkey@0"
I20260926 02:06:53.074237 20266 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:06:53.078197 20281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:53.079691 20266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:06:53.079895 20266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "132776399a2241e8b33be7693e69fd30"
format_stamp: "Formatted at 2026-09-26 02:06:53 on dist-test-slave-03rn"
server_key: "e8ea55f45f57b91756cb81d92d3cfbe9"
server_key_iv: "c36ccdbb1cb9d9a67b2ee993b72940d9"
server_key_version: "encryptionkey@0"
I20260926 02:06:53.080123 20266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:06:53.104327 20266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:06:53.109115 20266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:06:53.109426 20266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:06:53.112568 20266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:06:53.112775 20266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:53.112949 20266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:06:53.113037 20266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:06:53 dist-test-slave-03rn krb5kdc[20150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388413, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:06:53.136034 20289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:53.113009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:47785 (local address 127.18.159.190:45847)
0926 02:06:53.113523 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:53.113529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:53.114927 (+ 1398us) server_negotiation.cc:407] Connection header received
0926 02:06:53.117135 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:53.117145 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:53.117246 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:53.117373 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:53.119808 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:53.121135 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:53.123884 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:53.124561 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:53.124713 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:53.128861 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:53.128870 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:53.128878 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:53.128929 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:53.132693 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:53.133590 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:53.133596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:53.133600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:53.133734 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:53.134200 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:53.134205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:53.134209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:53.134507 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:53.134668 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:53.135430 (+ 762us) server_negotiation.cc:299] Negotiation successful
0926 02:06:53.135659 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":225,"threads_started":1}
I20260926 02:06:53.137404 20291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:53.113801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:45847 (local address 127.18.159.129:47785)
0926 02:06:53.114716 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:53.114774 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:53.116805 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:53.117664 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:53.117680 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:53.118596 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:06:53.119559 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:53.119597 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:53.121336 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:53.121345 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:53.123708 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:53.123722 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:53.124825 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:53.124833 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:53.125077 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:53.125822 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:06:53.125849 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:06:53.128571 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:06:53.132876 (+ 4305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:53.132887 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:53.132907 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:53.133433 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:53.133859 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:53.133864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:53.133868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:53.134080 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:53.134658 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:06:53.134666 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:06:53.135061 (+ 395us) client_negotiation.cc:769] Sending connection context
0926 02:06:53.135352 (+ 291us) client_negotiation.cc:240] Negotiation successful
0926 02:06:53.135943 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":403,"threads_started":1}
I20260926 02:06:53.167469 20289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:53.152363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:44353 (local address 127.18.159.190:45847)
0926 02:06:53.152538 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:53.152542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:53.153139 (+ 597us) server_negotiation.cc:407] Connection header received
0926 02:06:53.153355 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:53.153362 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:53.153430 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:53.153540 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:53.154513 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:53.155223 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:53.156683 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:53.157216 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:53.157356 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:53.160243 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:53.160254 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:53.160262 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:53.160317 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:53.164434 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:53.165337 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:53.165346 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:53.165351 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:53.165526 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:53.165897 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:53.165904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:53.165907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:53.166291 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:53.166454 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:53.167004 (+ 550us) server_negotiation.cc:299] Negotiation successful
0926 02:06:53.167211 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:06:53.168084 20357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:53.152375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:45847 (local address 127.18.159.129:44353)
0926 02:06:53.153004 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:53.153025 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:53.153220 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:53.153565 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:53.153570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:53.154060 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:06:53.154357 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:53.154366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:53.155361 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:53.155367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:53.156532 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:53.156546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:53.157467 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:53.157476 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:53.157748 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:53.158632 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:06:53.158664 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:06:53.159960 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:06:53.164628 (+ 4668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:53.164636 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:53.164642 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:53.165193 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:53.165647 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:53.165650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:53.165653 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:53.165772 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:53.166446 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:06:53.166458 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:06:53.166727 (+ 269us) client_negotiation.cc:769] Sending connection context
0926 02:06:53.166931 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 02:06:53.167414 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":344,"threads_started":1}
I20260926 02:06:53.177742 20266 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:42417
I20260926 02:06:53.177762 20401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:42417 every 8 connection(s)
I20260926 02:06:53.179064 20266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:06:53.181223 20402 heartbeater.cc:347] Connected to a master server at 127.18.159.190:45847
I20260926 02:06:53.181478 20402 heartbeater.cc:464] Registering TS with master...
I20260926 02:06:53.182180 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 20266
I20260926 02:06:53.182235 20402 heartbeater.cc:511] Master 127.18.159.190:45847 requested a full tablet report, sending...
I20260926 02:06:53.182410 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:06:53.183056 19070 external_mini_cluster.cc:1428] Setting key c2c07fde757d933d7ce1abf30716d1c3
I20260926 02:06:53.184751 20195 ts_manager.cc:194] Registered new tserver with Master: 132776399a2241e8b33be7693e69fd30 (127.18.159.129:42417)
I20260926 02:06:53.186269 20195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:47785
WARNING: no policy specified for kudu/127.18.159.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
W20260926 02:06:53.507931 19070 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.18.159.130: real 0.322s user 0.004s sys 0.004s
I20260926 02:06:53.508769 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:45847
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:38343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:06:53.846364 20406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:06:53.846693 20406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:06:53.846793 20406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:06:53.854691 20406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:06:53.854830 20406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:06:53.854916 20406 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:06:53.854998 20406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:06:53.865280 20406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:38343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.159.190:45847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:06:53.866727 20406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:06:53.868216 20406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:06:53.882899 20411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:53.882962 20412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:53.882992 20414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:06:53.883843 20406 server_base.cc:1034] running on GCE node
I20260926 02:06:53.884581 20406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:06:53.885637 20406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:06:53.886868 20406 hybrid_clock.cc:648] HybridClock initialized: now 1790388413886801 us; error 49 us; skew 500 ppm
I20260926 02:06:53.887229 20406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:06:53 dist-test-slave-03rn krb5kdc[20150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388413, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:06:53.890559 20406 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:06:53.892094 20406 webserver.cc:460] Webserver started at http://127.18.159.130:39019/ using document root <none> and password file <none>
I20260926 02:06:53.893019 20406 fs_manager.cc:359] Metadata directory not provided
I20260926 02:06:53.893160 20406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:06:53.893491 20406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:06:53.896690 20406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "40a92445911542ddae31959ca2cbe467"
format_stamp: "Formatted at 2026-09-26 02:06:53 on dist-test-slave-03rn"
server_key: "29cd0ac3a9296b3d6336e4707642fb6b"
server_key_iv: "0793b81a06f1fb2e201f23b3ce012a97"
server_key_version: "encryptionkey@0"
I20260926 02:06:53.897552 20406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "40a92445911542ddae31959ca2cbe467"
format_stamp: "Formatted at 2026-09-26 02:06:53 on dist-test-slave-03rn"
server_key: "29cd0ac3a9296b3d6336e4707642fb6b"
server_key_iv: "0793b81a06f1fb2e201f23b3ce012a97"
server_key_version: "encryptionkey@0"
I20260926 02:06:53.903000 20406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:06:53.906932 20421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:53.908555 20406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:06:53.908758 20406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "40a92445911542ddae31959ca2cbe467"
format_stamp: "Formatted at 2026-09-26 02:06:53 on dist-test-slave-03rn"
server_key: "29cd0ac3a9296b3d6336e4707642fb6b"
server_key_iv: "0793b81a06f1fb2e201f23b3ce012a97"
server_key_version: "encryptionkey@0"
I20260926 02:06:53.909017 20406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:06:53.929309 20406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:06:53.933938 20406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:06:53.934281 20406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:06:53.937173 20406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:06:53.937292 20406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:53.937425 20406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:06:53.937515 20406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:06:53 dist-test-slave-03rn krb5kdc[20150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388413, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:06:53.964993 20431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:53.938842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:45847 (local address 127.18.159.130:47085)
0926 02:06:53.939816 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:53.939859 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:53.942185 (+ 2326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:53.943455 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:53.943473 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:53.944487 (+ 1014us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:06:53.945525 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:53.945561 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:53.947087 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:53.947092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:53.949610 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:53.949631 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:53.950751 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:53.950761 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:53.951152 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:53.951955 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:06:53.951984 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:06:53.954900 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:06:53.959056 (+ 4156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:53.959070 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:53.959094 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:53.959870 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:53.960414 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:53.960419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:53.960423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:53.960787 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:53.961656 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:06:53.961669 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:06:53.962213 (+ 544us) client_negotiation.cc:769] Sending connection context
0926 02:06:53.962673 (+ 460us) client_negotiation.cc:240] Negotiation successful
0926 02:06:53.963373 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":400,"threads_started":1}
I20260926 02:06:53.965328 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:53.937992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:47085 (local address 127.18.159.190:45847)
0926 02:06:53.938540 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:53.938544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:53.940013 (+ 1469us) server_negotiation.cc:407] Connection header received
0926 02:06:53.942571 (+ 2558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:53.942576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:53.942646 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:53.942763 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:53.945753 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:53.946929 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:53.949847 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:53.950458 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:53.950640 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:53.955199 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:53.955205 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:53.955208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:53.955243 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:53.958850 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:53.960088 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:53.960093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:53.960099 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:53.960310 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:53.960975 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:53.960983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:53.960989 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:53.961415 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:53.961656 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:53.964409 (+ 2753us) server_negotiation.cc:299] Negotiation successful
0926 02:06:53.964772 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":336,"threads_started":1}
I20260926 02:06:53.988595 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:53.972882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:47661 (local address 127.18.159.190:45847)
0926 02:06:53.973141 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:53.973150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:53.973722 (+ 572us) server_negotiation.cc:407] Connection header received
0926 02:06:53.973867 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:53.973875 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:53.973981 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:53.974116 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:53.975053 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:53.976525 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:53.978426 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:53.979341 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:53.979521 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:53.981841 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:53.981846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:53.981850 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:53.981880 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:53.985434 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:53.986215 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:53.986222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:53.986226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:53.986362 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:53.986748 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:53.986756 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:53.986761 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:53.987183 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:53.987356 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:53.988074 (+ 718us) server_negotiation.cc:299] Negotiation successful
0926 02:06:53.988368 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 02:06:53.989066 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:53.972967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:45847 (local address 127.18.159.130:47661)
0926 02:06:53.973601 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:53.973622 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:53.973760 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:53.974156 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:53.974159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:53.974619 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:06:53.974904 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:53.974912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:53.976712 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:53.976719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:53.978257 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:53.978271 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:53.979558 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:53.979563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:53.979685 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:53.980363 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:06:53.980380 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:06:53.981609 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:06:53.985594 (+ 3985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:53.985600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:53.985604 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:53.986074 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:53.986476 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:53.986481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:53.986485 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:53.986631 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:53.987334 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:06:53.987343 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:06:53.987669 (+ 326us) client_negotiation.cc:769] Sending connection context
0926 02:06:53.987879 (+ 210us) client_negotiation.cc:240] Negotiation successful
0926 02:06:53.988383 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":338,"threads_started":1}
I20260926 02:06:54.002879 20406 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:46863
I20260926 02:06:54.002897 20541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:46863 every 8 connection(s)
I20260926 02:06:54.004354 20406 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:06:54.007537 20542 heartbeater.cc:347] Connected to a master server at 127.18.159.190:45847
I20260926 02:06:54.007822 20542 heartbeater.cc:464] Registering TS with master...
I20260926 02:06:54.008589 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 20406
I20260926 02:06:54.008788 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:06:54.008790 20542 heartbeater.cc:511] Master 127.18.159.190:45847 requested a full tablet report, sending...
I20260926 02:06:54.009248 19070 external_mini_cluster.cc:1428] Setting key 03e720e983034117491cce5a5c68d141
I20260926 02:06:54.010587 20195 ts_manager.cc:194] Registered new tserver with Master: 40a92445911542ddae31959ca2cbe467 (127.18.159.130:46863)
I20260926 02:06:54.011652 20195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:47085
WARNING: no policy specified for kudu/127.18.159.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
I20260926 02:06:54.052727 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:45847
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:38343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:06:54.189509 20402 heartbeater.cc:503] Master 127.18.159.190:45847 was elected leader, sending a full tablet report...
W20260926 02:06:54.402231 20546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:06:54.402537 20546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:06:54.402635 20546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:06:54.410145 20546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:06:54.410269 20546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:06:54.410353 20546 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:06:54.410435 20546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:06:54.421144 20546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:38343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.159.190:45847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:06:54.422412 20546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:06:54.423768 20546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:06:54.438372 20552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:54.438292 20554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:54.438274 20551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:06:54.438771 20546 server_base.cc:1034] running on GCE node
I20260926 02:06:54.439601 20546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:06:54.440748 20546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:06:54.442065 20546 hybrid_clock.cc:648] HybridClock initialized: now 1790388414441967 us; error 72 us; skew 500 ppm
I20260926 02:06:54.442432 20546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:06:54 dist-test-slave-03rn krb5kdc[20150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388414, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:06:54.445820 20546 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:06:54.447414 20546 webserver.cc:460] Webserver started at http://127.18.159.131:32839/ using document root <none> and password file <none>
I20260926 02:06:54.448115 20546 fs_manager.cc:359] Metadata directory not provided
I20260926 02:06:54.448272 20546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:06:54.448561 20546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:06:54.451841 20546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "dabdcdc9ec7a457a95e08b67b752e2cd"
format_stamp: "Formatted at 2026-09-26 02:06:54 on dist-test-slave-03rn"
server_key: "4ce368470a434b998e3147c6aa2dcd9d"
server_key_iv: "7613a09bcd29ac9e9375346663a90456"
server_key_version: "encryptionkey@0"
I20260926 02:06:54.452728 20546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "dabdcdc9ec7a457a95e08b67b752e2cd"
format_stamp: "Formatted at 2026-09-26 02:06:54 on dist-test-slave-03rn"
server_key: "4ce368470a434b998e3147c6aa2dcd9d"
server_key_iv: "7613a09bcd29ac9e9375346663a90456"
server_key_version: "encryptionkey@0"
I20260926 02:06:54.458250 20546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:06:54.462095 20561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:54.463701 20546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:06:54.463856 20546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "dabdcdc9ec7a457a95e08b67b752e2cd"
format_stamp: "Formatted at 2026-09-26 02:06:54 on dist-test-slave-03rn"
server_key: "4ce368470a434b998e3147c6aa2dcd9d"
server_key_iv: "7613a09bcd29ac9e9375346663a90456"
server_key_version: "encryptionkey@0"
I20260926 02:06:54.464064 20546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:06:54.476429 20546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:06:54.481343 20546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:06:54.481660 20546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:06:54.484556 20546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:06:54.484642 20546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:54.484827 20546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:06:54.484906 20546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:06:54 dist-test-slave-03rn krb5kdc[20150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388414, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:06:54.515154 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:54.485312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:56451 (local address 127.18.159.190:45847)
0926 02:06:54.485501 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:54.485507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:54.487051 (+ 1544us) server_negotiation.cc:407] Connection header received
0926 02:06:54.490326 (+ 3275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:54.490337 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:54.490445 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:54.490549 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:54.494304 (+ 3755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.495878 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.499546 (+ 3668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.500305 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.500446 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:54.506005 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:54.506017 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:54.506025 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:54.506073 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:54.510455 (+ 4382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:54.511772 (+ 1317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:54.511782 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:54.511785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:54.511947 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:54.512692 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:54.512702 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:54.512707 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:54.513167 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:54.513365 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:54.514524 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0926 02:06:54.514779 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:06:54.516528 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:54.485934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:45847 (local address 127.18.159.131:56451)
0926 02:06:54.486828 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:54.486887 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:54.489853 (+ 2966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:54.491297 (+ 1444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:54.491318 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:54.492747 (+ 1429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:06:54.493980 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:54.494017 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.496212 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.496221 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:54.499278 (+ 3057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:54.499306 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.500767 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.500777 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:54.501224 (+ 447us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:54.502245 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:06:54.502274 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:06:54.505572 (+ 3298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:06:54.510727 (+ 5155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:54.510744 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:54.510775 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:54.511531 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:54.512079 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:54.512084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:54.512098 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:54.512502 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:54.513364 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:06:54.513377 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:06:54.513987 (+ 610us) client_negotiation.cc:769] Sending connection context
0926 02:06:54.514410 (+ 423us) client_negotiation.cc:240] Negotiation successful
0926 02:06:54.515012 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":331,"threads_started":1}
I20260926 02:06:54.538975 20629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:54.522489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:45847 (local address 127.18.159.131:54553)
0926 02:06:54.523268 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:54.523299 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:54.523497 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:54.523926 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:54.523933 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:54.524801 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:06:54.525416 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:54.525433 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.526848 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.526853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:54.527891 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:54.527902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.528828 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.528841 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:54.529116 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:54.529880 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:06:54.529913 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:06:54.531319 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:06:54.535447 (+ 4128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:54.535457 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:54.535464 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:54.536397 (+ 933us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:54.537000 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:54.537005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:54.537010 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:54.537238 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:54.537951 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:06:54.537954 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:06:54.538117 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 02:06:54.538256 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 02:06:54.538540 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":380,"threads_started":1}
I20260926 02:06:54.539142 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:54.522483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:54553 (local address 127.18.159.190:45847)
0926 02:06:54.522728 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:54.522738 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:54.523411 (+ 673us) server_negotiation.cc:407] Connection header received
0926 02:06:54.523671 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:54.523680 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:54.523767 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:54.523879 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:54.525709 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.526707 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.528045 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.528602 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.528756 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:54.531629 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:54.531636 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:54.531640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:54.531680 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:54.535238 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:54.536597 (+ 1359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:54.536606 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:54.536612 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:54.536845 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:54.537391 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:54.537399 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:54.537404 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:54.537806 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:54.537970 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:54.538395 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 02:06:54.538686 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:06:54.548812 20546 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:41629
I20260926 02:06:54.548831 20680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:41629 every 8 connection(s)
I20260926 02:06:54.550237 20546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:06:54.552769 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 20546
I20260926 02:06:54.552835 20681 heartbeater.cc:347] Connected to a master server at 127.18.159.190:45847
I20260926 02:06:54.552964 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:06:54.553165 20681 heartbeater.cc:464] Registering TS with master...
I20260926 02:06:54.553431 19070 external_mini_cluster.cc:1428] Setting key 66c9426d206961b3a41b6dec8007e7b7
I20260926 02:06:54.553941 20681 heartbeater.cc:511] Master 127.18.159.190:45847 requested a full tablet report, sending...
I20260926 02:06:54.555579 20195 ts_manager.cc:194] Registered new tserver with Master: dabdcdc9ec7a457a95e08b67b752e2cd (127.18.159.131:41629)
I20260926 02:06:54.556572 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:06:54.556787 20195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:56451
Sep 26 02:06:54 dist-test-slave-03rn krb5kdc[20150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388414, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 02:06:54 dist-test-slave-03rn krb5kdc[20150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388414, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:06:54.609135 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:54.589925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.18.159.190:45847)
0926 02:06:54.590121 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:54.590126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:54.590548 (+ 422us) server_negotiation.cc:407] Connection header received
0926 02:06:54.590701 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:54.590705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:54.590773 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:54.590872 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:54.592698 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.593431 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.594880 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.595297 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.595403 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:54.600038 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:54.600050 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:54.600057 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:54.600116 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:54.604591 (+ 4475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:54.606011 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:54.606019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:54.606024 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:54.606397 (+ 373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:54.606908 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:54.606916 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:54.606921 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:54.607385 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:54.607744 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:54.608332 (+ 588us) server_negotiation.cc:299] Negotiation successful
0926 02:06:54.608659 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:06:54.615684 20195 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:06:54.625818 20195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:06:54.673401 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:54.643360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:42417 (local address 127.0.0.1:54208)
0926 02:06:54.644150 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:54.644235 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:54.644436 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:54.659772 (+ 15336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:54.659780 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:54.659844 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:06:54.660444 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:54.660466 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.663873 (+ 3407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.663881 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:54.666551 (+ 2670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:54.666570 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.671892 (+ 5322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.671899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:54.672200 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:54.672581 (+ 381us) client_negotiation.cc:769] Sending connection context
0926 02:06:54.672726 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 02:06:54.672952 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":295,"threads_started":1}
I20260926 02:06:54.674235 20699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:54.645046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.18.159.130:46863)
0926 02:06:54.661762 (+ 16716us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:54.661777 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:54.664943 (+ 3166us) server_negotiation.cc:407] Connection header received
0926 02:06:54.666076 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:54.666082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:54.666280 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:54.666513 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:06:54.667266 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.669068 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.671037 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.672154 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.672354 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:54.672558 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:54.673785 (+ 1227us) server_negotiation.cc:299] Negotiation successful
0926 02:06:54.673948 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16422,"thread_start_us":18653,"threads_started":1}
I20260926 02:06:54.674221 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:54.643578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54208 (local address 127.18.159.129:42417)
0926 02:06:54.658831 (+ 15253us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:54.658844 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:54.658891 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:06:54.659055 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:54.659065 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:54.659426 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:54.659759 (+ 333us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:06:54.661724 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.663702 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.669893 (+ 6191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.671633 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.671843 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:54.672051 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:54.673707 (+ 1656us) server_negotiation.cc:299] Negotiation successful
0926 02:06:54.673911 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15041,"thread_start_us":14886,"threads_started":1}
I20260926 02:06:54.673498 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:54.644254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:46863 (local address 127.0.0.1:60986)
0926 02:06:54.664727 (+ 20473us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:54.664761 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:54.664966 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:54.666520 (+ 1554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:54.666525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:54.666549 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:06:54.667036 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:54.667050 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.669219 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.669227 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:54.670790 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:54.670807 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.672708 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.672717 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:54.672879 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:54.672910 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 02:06:54.672992 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 02:06:54.673123 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20215,"mutex_wait_us":4,"thread_start_us":19972,"threads_started":1}
I20260926 02:06:54.674813 20697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:54.644740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:41629 (local address 127.0.0.1:47978)
0926 02:06:54.665716 (+ 20976us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:54.665732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:54.665837 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:54.666870 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:54.666874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:54.666915 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:06:54.667364 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:54.667380 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.669806 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.669815 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:54.672054 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:54.672073 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.673980 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.673986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:54.674168 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:54.674205 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 02:06:54.674300 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 02:06:54.674440 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20846,"thread_start_us":20663,"threads_started":1}
I20260926 02:06:54.675135 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:54.645028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47978 (local address 127.18.159.131:41629)
0926 02:06:54.656422 (+ 11394us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:54.656434 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:54.665844 (+ 9410us) server_negotiation.cc:407] Connection header received
0926 02:06:54.665973 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:54.665982 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:54.666283 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:54.666516 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:06:54.667630 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.669630 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.672260 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.673754 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.673980 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:54.674158 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:54.674351 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 02:06:54.674522 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11143,"thread_start_us":11190,"threads_started":1}
I20260926 02:06:54.678591 20472 tablet_service.cc:1467] Processing CreateTablet for tablet 29f73547998a4a9b9ce0ea9df67be085 (DEFAULT_TABLE table=test-table [id=b1cbc523c81545ddb22b4b04f954a10b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:06:54.678494 20611 tablet_service.cc:1467] Processing CreateTablet for tablet 29f73547998a4a9b9ce0ea9df67be085 (DEFAULT_TABLE table=test-table [id=b1cbc523c81545ddb22b4b04f954a10b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:06:54.678792 20332 tablet_service.cc:1467] Processing CreateTablet for tablet 29f73547998a4a9b9ce0ea9df67be085 (DEFAULT_TABLE table=test-table [id=b1cbc523c81545ddb22b4b04f954a10b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:06:54.680115 20332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f73547998a4a9b9ce0ea9df67be085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:06:54.680137 20611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f73547998a4a9b9ce0ea9df67be085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:06:54.680135 20472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f73547998a4a9b9ce0ea9df67be085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:06:54.692363 20700 tablet_bootstrap.cc:492] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd: Bootstrap starting.
I20260926 02:06:54.696714 20700 tablet_bootstrap.cc:654] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd: Neither blocks nor log segments found. Creating new log.
I20260926 02:06:54.698015 20700 log.cc:824] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd: Log is configured to *not* fsync() on all Append() calls
I20260926 02:06:54.701472 20700 tablet_bootstrap.cc:492] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd: No bootstrap required, opened a new log
I20260926 02:06:54.701798 20700 ts_tablet_manager.cc:1397] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 02:06:54.704991 20702 tablet_bootstrap.cc:492] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467: Bootstrap starting.
I20260926 02:06:54.705773 20701 tablet_bootstrap.cc:492] T 29f73547998a4a9b9ce0ea9df67be085 P 132776399a2241e8b33be7693e69fd30: Bootstrap starting.
I20260926 02:06:54.709332 20701 tablet_bootstrap.cc:654] T 29f73547998a4a9b9ce0ea9df67be085 P 132776399a2241e8b33be7693e69fd30: Neither blocks nor log segments found. Creating new log.
I20260926 02:06:54.709398 20702 tablet_bootstrap.cc:654] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467: Neither blocks nor log segments found. Creating new log.
I20260926 02:06:54.710616 20701 log.cc:824] T 29f73547998a4a9b9ce0ea9df67be085 P 132776399a2241e8b33be7693e69fd30: Log is configured to *not* fsync() on all Append() calls
I20260926 02:06:54.710728 20702 log.cc:824] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467: Log is configured to *not* fsync() on all Append() calls
I20260926 02:06:54.712507 20700 raft_consensus.cc:348] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } }
I20260926 02:06:54.713091 20700 raft_consensus.cc:374] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:06:54.713194 20700 raft_consensus.cc:729] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabdcdc9ec7a457a95e08b67b752e2cd, State: Initialized, Role: FOLLOWER
I20260926 02:06:54.713888 20700 consensus_queue.cc:260] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } }
I20260926 02:06:54.714726 20701 tablet_bootstrap.cc:492] T 29f73547998a4a9b9ce0ea9df67be085 P 132776399a2241e8b33be7693e69fd30: No bootstrap required, opened a new log
I20260926 02:06:54.715058 20701 ts_tablet_manager.cc:1397] T 29f73547998a4a9b9ce0ea9df67be085 P 132776399a2241e8b33be7693e69fd30: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260926 02:06:54.715096 20681 heartbeater.cc:503] Master 127.18.159.190:45847 was elected leader, sending a full tablet report...
I20260926 02:06:54.715711 20700 ts_tablet_manager.cc:1428] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260926 02:06:54.718183 20702 tablet_bootstrap.cc:492] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467: No bootstrap required, opened a new log
I20260926 02:06:54.718523 20702 ts_tablet_manager.cc:1397] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260926 02:06:54.724100 20701 raft_consensus.cc:348] T 29f73547998a4a9b9ce0ea9df67be085 P 132776399a2241e8b33be7693e69fd30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } }
I20260926 02:06:54.724627 20701 raft_consensus.cc:374] T 29f73547998a4a9b9ce0ea9df67be085 P 132776399a2241e8b33be7693e69fd30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:06:54.724754 20701 raft_consensus.cc:729] T 29f73547998a4a9b9ce0ea9df67be085 P 132776399a2241e8b33be7693e69fd30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 132776399a2241e8b33be7693e69fd30, State: Initialized, Role: FOLLOWER
I20260926 02:06:54.725342 20701 consensus_queue.cc:260] T 29f73547998a4a9b9ce0ea9df67be085 P 132776399a2241e8b33be7693e69fd30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } }
I20260926 02:06:54.727052 20701 ts_tablet_manager.cc:1428] T 29f73547998a4a9b9ce0ea9df67be085 P 132776399a2241e8b33be7693e69fd30: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260926 02:06:54.728564 20702 raft_consensus.cc:348] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } }
I20260926 02:06:54.729072 20702 raft_consensus.cc:374] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:06:54.729219 20702 raft_consensus.cc:729] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a92445911542ddae31959ca2cbe467, State: Initialized, Role: FOLLOWER
I20260926 02:06:54.729938 20702 consensus_queue.cc:260] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } }
I20260926 02:06:54.731145 20542 heartbeater.cc:503] Master 127.18.159.190:45847 was elected leader, sending a full tablet report...
I20260926 02:06:54.731902 20702 ts_tablet_manager.cc:1428] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
W20260926 02:06:54.756929 20543 tablet.cc:2369] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:06:54.762223 20708 raft_consensus.cc:482] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:06:54.762524 20708 raft_consensus.cc:504] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } }
I20260926 02:06:54.764245 20708 leader_election.cc:288] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 132776399a2241e8b33be7693e69fd30 (127.18.159.129:42417), dabdcdc9ec7a457a95e08b67b752e2cd (127.18.159.131:41629)
I20260926 02:06:54.772454 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:54.764781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:37895 (local address 127.18.159.129:42417)
0926 02:06:54.765067 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:54.765077 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:54.765509 (+ 432us) server_negotiation.cc:407] Connection header received
0926 02:06:54.765766 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:54.765771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:54.765861 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:54.765927 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:06:54.766821 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.768460 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.770406 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.771359 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.771493 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:54.771547 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:54.771995 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 02:06:54.772205 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:06:54.772580 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:54.764644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:42417 (local address 127.18.159.130:37895)
0926 02:06:54.765356 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:54.765372 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:54.765559 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:54.766122 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:54.766126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:54.766244 (+ 118us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:06:54.766645 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:54.766664 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.768613 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.768622 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:54.770252 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:54.770264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.771606 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.771613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:54.771786 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:54.771824 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 02:06:54.771949 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 02:06:54.772117 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":29,"spinlock_wait_cycles":43264,"thread_start_us":423,"threads_started":1}
I20260926 02:06:54.772918 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:54.765752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:51951 (local address 127.18.159.131:41629)
0926 02:06:54.765931 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:54.765937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:54.766192 (+ 255us) server_negotiation.cc:407] Connection header received
0926 02:06:54.766329 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:54.766336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:54.766410 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:54.766509 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:06:54.767700 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.768760 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.770799 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.771837 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.772004 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:54.772069 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:54.772457 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 02:06:54.772655 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:06:54.773370 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:54.765494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:41629 (local address 127.18.159.130:51951)
0926 02:06:54.766084 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:54.766099 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:54.766213 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:54.766565 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:54.766569 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:54.766590 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:06:54.767076 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:54.767087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.768962 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.768966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:54.770576 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:54.770590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.772071 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.772078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:54.772285 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:54.772321 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:06:54.772419 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 02:06:54.773012 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":28,"thread_start_us":301,"threads_started":1}
I20260926 02:06:54.773768 20356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f73547998a4a9b9ce0ea9df67be085" candidate_uuid: "40a92445911542ddae31959ca2cbe467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "132776399a2241e8b33be7693e69fd30" is_pre_election: true
I20260926 02:06:54.774363 20356 raft_consensus.cc:2454] T 29f73547998a4a9b9ce0ea9df67be085 P 132776399a2241e8b33be7693e69fd30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40a92445911542ddae31959ca2cbe467 in term 0.
I20260926 02:06:54.774363 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f73547998a4a9b9ce0ea9df67be085" candidate_uuid: "40a92445911542ddae31959ca2cbe467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" is_pre_election: true
I20260926 02:06:54.774941 20636 raft_consensus.cc:2454] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40a92445911542ddae31959ca2cbe467 in term 0.
I20260926 02:06:54.775175 20422 leader_election.cc:302] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 132776399a2241e8b33be7693e69fd30, 40a92445911542ddae31959ca2cbe467; no voters:
I20260926 02:06:54.775791 20708 raft_consensus.cc:2781] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:06:54.776028 20708 raft_consensus.cc:482] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:06:54.776211 20708 raft_consensus.cc:3037] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:06:54.778551 20708 raft_consensus.cc:504] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } }
I20260926 02:06:54.779239 20708 leader_election.cc:288] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 [CANDIDATE]: Term 1 election: Requested vote from peers 132776399a2241e8b33be7693e69fd30 (127.18.159.129:42417), dabdcdc9ec7a457a95e08b67b752e2cd (127.18.159.131:41629)
I20260926 02:06:54.779871 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f73547998a4a9b9ce0ea9df67be085" candidate_uuid: "40a92445911542ddae31959ca2cbe467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd"
I20260926 02:06:54.779835 20356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f73547998a4a9b9ce0ea9df67be085" candidate_uuid: "40a92445911542ddae31959ca2cbe467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "132776399a2241e8b33be7693e69fd30"
I20260926 02:06:54.780084 20356 raft_consensus.cc:3037] T 29f73547998a4a9b9ce0ea9df67be085 P 132776399a2241e8b33be7693e69fd30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:06:54.780084 20636 raft_consensus.cc:3037] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:06:54.783907 20636 raft_consensus.cc:2454] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40a92445911542ddae31959ca2cbe467 in term 1.
I20260926 02:06:54.783907 20356 raft_consensus.cc:2454] T 29f73547998a4a9b9ce0ea9df67be085 P 132776399a2241e8b33be7693e69fd30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40a92445911542ddae31959ca2cbe467 in term 1.
I20260926 02:06:54.784426 20422 leader_election.cc:302] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 132776399a2241e8b33be7693e69fd30, 40a92445911542ddae31959ca2cbe467; no voters:
I20260926 02:06:54.784801 20708 raft_consensus.cc:2781] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:06:54.785315 20708 raft_consensus.cc:686] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 [term 1 LEADER]: Becoming Leader. State: Replica: 40a92445911542ddae31959ca2cbe467, State: Running, Role: LEADER
I20260926 02:06:54.785879 20708 consensus_queue.cc:237] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } }
I20260926 02:06:54.791111 20195 catalog_manager.cc:5495] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40a92445911542ddae31959ca2cbe467 (127.18.159.130). New cstate: current_term: 1 leader_uuid: "40a92445911542ddae31959ca2cbe467" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } health_report { overall_health: HEALTHY } } }
W20260926 02:06:54.802263 20682 tablet.cc:2369] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:06:54.832121 20699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:54.825602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32770 (local address 127.18.159.130:46863)
0926 02:06:54.825817 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:54.825822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:54.825985 (+ 163us) server_negotiation.cc:407] Connection header received
0926 02:06:54.826370 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:54.826374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:54.826459 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:54.826617 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:06:54.826982 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.827728 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.829198 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:54.829628 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:54.829771 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:54.830346 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:06:54.830935 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:06:54.831103 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:54.831266 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 02:06:54.831357 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:06:54.841375 20636 raft_consensus.cc:1261] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd [term 1 FOLLOWER]: Refusing update from remote peer 40a92445911542ddae31959ca2cbe467: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:06:54.841375 20356 raft_consensus.cc:1261] T 29f73547998a4a9b9ce0ea9df67be085 P 132776399a2241e8b33be7693e69fd30 [term 1 FOLLOWER]: Refusing update from remote peer 40a92445911542ddae31959ca2cbe467: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:06:54.842566 20711 consensus_queue.cc:1035] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 [LEADER]: Connected to new peer: Peer: permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:06:54.842938 20708 consensus_queue.cc:1035] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:06:54.858560 20716 mvcc.cc:205] Tried to move back new op lower bound from 7333430947177967616 to 7333430946967027712. Current Snapshot: MvccSnapshot[applied={T|T < 7333430947177967616}]
I20260926 02:06:54.879395 20195 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:06:54.879879 20195 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:06:54.884267 20195 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133: Sending DeleteTablet for 3 replicas of tablet 29f73547998a4a9b9ce0ea9df67be085
I20260926 02:06:54.885605 20472 tablet_service.cc:1514] Processing DeleteTablet for tablet 29f73547998a4a9b9ce0ea9df67be085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:06:54 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:60986
I20260926 02:06:54.885583 20611 tablet_service.cc:1514] Processing DeleteTablet for tablet 29f73547998a4a9b9ce0ea9df67be085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:06:54 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:47978
I20260926 02:06:54.885640 20332 tablet_service.cc:1514] Processing DeleteTablet for tablet 29f73547998a4a9b9ce0ea9df67be085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:06:54 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:54208
I20260926 02:06:54.888410 20195 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:06:54.889325 20195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:06:54.896840 20724 tablet_replica.cc:330] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd: stopping tablet replica
I20260926 02:06:54.897526 20724 raft_consensus.cc:2229] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:06:54.898212 20724 raft_consensus.cc:2258] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:06:54.899055 20725 tablet_replica.cc:330] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467: stopping tablet replica
I20260926 02:06:54.899278 20726 tablet_replica.cc:330] T 29f73547998a4a9b9ce0ea9df67be085 P 132776399a2241e8b33be7693e69fd30: stopping tablet replica
I20260926 02:06:54.899515 20725 raft_consensus.cc:2229] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:06:54.900106 20726 raft_consensus.cc:2229] T 29f73547998a4a9b9ce0ea9df67be085 P 132776399a2241e8b33be7693e69fd30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:06:54.900288 20725 raft_consensus.cc:2258] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:06:54.900660 20726 raft_consensus.cc:2258] T 29f73547998a4a9b9ce0ea9df67be085 P 132776399a2241e8b33be7693e69fd30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:06:54.902314 20724 ts_tablet_manager.cc:1905] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:06:54.902673 20332 tablet_service.cc:1467] Processing CreateTablet for tablet 6511bccfd3974779ad5750ab4549c6ae (DEFAULT_TABLE table=test-table [id=16dbf1dbf71e4e02a3a603c08c145f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:06:54.902941 20472 tablet_service.cc:1467] Processing CreateTablet for tablet 6511bccfd3974779ad5750ab4549c6ae (DEFAULT_TABLE table=test-table [id=16dbf1dbf71e4e02a3a603c08c145f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:06:54.903412 20472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6511bccfd3974779ad5750ab4549c6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:06:54.903911 20332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6511bccfd3974779ad5750ab4549c6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:06:54.908599 20611 tablet_service.cc:1467] Processing CreateTablet for tablet 6511bccfd3974779ad5750ab4549c6ae (DEFAULT_TABLE table=test-table [id=16dbf1dbf71e4e02a3a603c08c145f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:06:54.909121 20611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6511bccfd3974779ad5750ab4549c6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:06:54.909754 20724 ts_tablet_manager.cc:1918] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:06:54.909950 20724 log.cc:1196] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/29f73547998a4a9b9ce0ea9df67be085
I20260926 02:06:54.910632 20724 ts_tablet_manager.cc:1939] T 29f73547998a4a9b9ce0ea9df67be085 P dabdcdc9ec7a457a95e08b67b752e2cd: Deleting consensus metadata
I20260926 02:06:54.914031 20701 tablet_bootstrap.cc:492] T 6511bccfd3974779ad5750ab4549c6ae P 132776399a2241e8b33be7693e69fd30: Bootstrap starting.
I20260926 02:06:54.914116 20725 ts_tablet_manager.cc:1905] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:06:54.914518 20726 ts_tablet_manager.cc:1905] T 29f73547998a4a9b9ce0ea9df67be085 P 132776399a2241e8b33be7693e69fd30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:06:54.915760 20701 tablet_bootstrap.cc:654] T 6511bccfd3974779ad5750ab4549c6ae P 132776399a2241e8b33be7693e69fd30: Neither blocks nor log segments found. Creating new log.
I20260926 02:06:54.922874 20725 ts_tablet_manager.cc:1918] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:06:54.923110 20725 log.cc:1196] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/29f73547998a4a9b9ce0ea9df67be085
I20260926 02:06:54.923969 20700 tablet_bootstrap.cc:492] T 6511bccfd3974779ad5750ab4549c6ae P dabdcdc9ec7a457a95e08b67b752e2cd: Bootstrap starting.
I20260926 02:06:54.925776 20700 tablet_bootstrap.cc:654] T 6511bccfd3974779ad5750ab4549c6ae P dabdcdc9ec7a457a95e08b67b752e2cd: Neither blocks nor log segments found. Creating new log.
I20260926 02:06:54.926992 20702 tablet_bootstrap.cc:492] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467: Bootstrap starting.
I20260926 02:06:54.929761 20702 tablet_bootstrap.cc:654] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467: Neither blocks nor log segments found. Creating new log.
I20260926 02:06:54.930377 20179 catalog_manager.cc:4841] TS dabdcdc9ec7a457a95e08b67b752e2cd (127.18.159.131:41629): tablet 29f73547998a4a9b9ce0ea9df67be085 (table test-table [id=b1cbc523c81545ddb22b4b04f954a10b]) successfully deleted
I20260926 02:06:54.931527 20725 ts_tablet_manager.cc:1939] T 29f73547998a4a9b9ce0ea9df67be085 P 40a92445911542ddae31959ca2cbe467: Deleting consensus metadata
I20260926 02:06:54.932082 20726 ts_tablet_manager.cc:1918] T 29f73547998a4a9b9ce0ea9df67be085 P 132776399a2241e8b33be7693e69fd30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:06:54.932979 20726 log.cc:1196] T 29f73547998a4a9b9ce0ea9df67be085 P 132776399a2241e8b33be7693e69fd30: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/29f73547998a4a9b9ce0ea9df67be085
I20260926 02:06:54.933991 20182 catalog_manager.cc:4841] TS 40a92445911542ddae31959ca2cbe467 (127.18.159.130:46863): tablet 29f73547998a4a9b9ce0ea9df67be085 (table test-table [id=b1cbc523c81545ddb22b4b04f954a10b]) successfully deleted
I20260926 02:06:54.933604 20726 ts_tablet_manager.cc:1939] T 29f73547998a4a9b9ce0ea9df67be085 P 132776399a2241e8b33be7693e69fd30: Deleting consensus metadata
I20260926 02:06:54.935225 20700 tablet_bootstrap.cc:492] T 6511bccfd3974779ad5750ab4549c6ae P dabdcdc9ec7a457a95e08b67b752e2cd: No bootstrap required, opened a new log
I20260926 02:06:54.935299 20702 tablet_bootstrap.cc:492] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467: No bootstrap required, opened a new log
I20260926 02:06:54.935367 20700 ts_tablet_manager.cc:1397] T 6511bccfd3974779ad5750ab4549c6ae P dabdcdc9ec7a457a95e08b67b752e2cd: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260926 02:06:54.935444 20702 ts_tablet_manager.cc:1397] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260926 02:06:54.936208 20700 raft_consensus.cc:348] T 6511bccfd3974779ad5750ab4549c6ae P dabdcdc9ec7a457a95e08b67b752e2cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } }
I20260926 02:06:54.936450 20700 raft_consensus.cc:374] T 6511bccfd3974779ad5750ab4549c6ae P dabdcdc9ec7a457a95e08b67b752e2cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:06:54.936394 20702 raft_consensus.cc:348] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } }
I20260926 02:06:54.936539 20700 raft_consensus.cc:729] T 6511bccfd3974779ad5750ab4549c6ae P dabdcdc9ec7a457a95e08b67b752e2cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabdcdc9ec7a457a95e08b67b752e2cd, State: Initialized, Role: FOLLOWER
I20260926 02:06:54.936621 20702 raft_consensus.cc:374] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:06:54.936718 20702 raft_consensus.cc:729] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a92445911542ddae31959ca2cbe467, State: Initialized, Role: FOLLOWER
I20260926 02:06:54.936944 20701 tablet_bootstrap.cc:492] T 6511bccfd3974779ad5750ab4549c6ae P 132776399a2241e8b33be7693e69fd30: No bootstrap required, opened a new log
I20260926 02:06:54.936800 20700 consensus_queue.cc:260] T 6511bccfd3974779ad5750ab4549c6ae P dabdcdc9ec7a457a95e08b67b752e2cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } }
I20260926 02:06:54.937091 20701 ts_tablet_manager.cc:1397] T 6511bccfd3974779ad5750ab4549c6ae P 132776399a2241e8b33be7693e69fd30: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.007s
I20260926 02:06:54.937028 20702 consensus_queue.cc:260] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } }
I20260926 02:06:54.937376 20179 catalog_manager.cc:4841] TS 132776399a2241e8b33be7693e69fd30 (127.18.159.129:42417): tablet 29f73547998a4a9b9ce0ea9df67be085 (table test-table [id=b1cbc523c81545ddb22b4b04f954a10b]) successfully deleted
I20260926 02:06:54.937637 20702 ts_tablet_manager.cc:1428] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:06:54.937877 20701 raft_consensus.cc:348] T 6511bccfd3974779ad5750ab4549c6ae P 132776399a2241e8b33be7693e69fd30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } }
I20260926 02:06:54.938099 20701 raft_consensus.cc:374] T 6511bccfd3974779ad5750ab4549c6ae P 132776399a2241e8b33be7693e69fd30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:06:54.938208 20701 raft_consensus.cc:729] T 6511bccfd3974779ad5750ab4549c6ae P 132776399a2241e8b33be7693e69fd30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 132776399a2241e8b33be7693e69fd30, State: Initialized, Role: FOLLOWER
I20260926 02:06:54.938494 20701 consensus_queue.cc:260] T 6511bccfd3974779ad5750ab4549c6ae P 132776399a2241e8b33be7693e69fd30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } }
I20260926 02:06:54.939124 20700 ts_tablet_manager.cc:1428] T 6511bccfd3974779ad5750ab4549c6ae P dabdcdc9ec7a457a95e08b67b752e2cd: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260926 02:06:54.939123 20701 ts_tablet_manager.cc:1428] T 6511bccfd3974779ad5750ab4549c6ae P 132776399a2241e8b33be7693e69fd30: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 02:06:55.063730 20719 raft_consensus.cc:482] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:06:55.064033 20719 raft_consensus.cc:504] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } }
I20260926 02:06:55.065521 20719 leader_election.cc:288] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 132776399a2241e8b33be7693e69fd30 (127.18.159.129:42417), dabdcdc9ec7a457a95e08b67b752e2cd (127.18.159.131:41629)
I20260926 02:06:55.066227 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6511bccfd3974779ad5750ab4549c6ae" candidate_uuid: "40a92445911542ddae31959ca2cbe467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" is_pre_election: true
I20260926 02:06:55.066269 20356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6511bccfd3974779ad5750ab4549c6ae" candidate_uuid: "40a92445911542ddae31959ca2cbe467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "132776399a2241e8b33be7693e69fd30" is_pre_election: true
I20260926 02:06:55.066490 20636 raft_consensus.cc:2454] T 6511bccfd3974779ad5750ab4549c6ae P dabdcdc9ec7a457a95e08b67b752e2cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40a92445911542ddae31959ca2cbe467 in term 0.
I20260926 02:06:55.066560 20356 raft_consensus.cc:2454] T 6511bccfd3974779ad5750ab4549c6ae P 132776399a2241e8b33be7693e69fd30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40a92445911542ddae31959ca2cbe467 in term 0.
I20260926 02:06:55.066980 20422 leader_election.cc:302] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 132776399a2241e8b33be7693e69fd30, 40a92445911542ddae31959ca2cbe467; no voters:
I20260926 02:06:55.067304 20719 raft_consensus.cc:2781] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:06:55.067435 20719 raft_consensus.cc:482] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:06:55.067584 20719 raft_consensus.cc:3037] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:06:55.070304 20719 raft_consensus.cc:504] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } }
I20260926 02:06:55.071024 20719 leader_election.cc:288] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 [CANDIDATE]: Term 1 election: Requested vote from peers 132776399a2241e8b33be7693e69fd30 (127.18.159.129:42417), dabdcdc9ec7a457a95e08b67b752e2cd (127.18.159.131:41629)
I20260926 02:06:55.071664 20356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6511bccfd3974779ad5750ab4549c6ae" candidate_uuid: "40a92445911542ddae31959ca2cbe467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "132776399a2241e8b33be7693e69fd30"
I20260926 02:06:55.071738 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6511bccfd3974779ad5750ab4549c6ae" candidate_uuid: "40a92445911542ddae31959ca2cbe467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd"
I20260926 02:06:55.071910 20356 raft_consensus.cc:3037] T 6511bccfd3974779ad5750ab4549c6ae P 132776399a2241e8b33be7693e69fd30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:06:55.071981 20636 raft_consensus.cc:3037] T 6511bccfd3974779ad5750ab4549c6ae P dabdcdc9ec7a457a95e08b67b752e2cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:06:55.074681 20356 raft_consensus.cc:2454] T 6511bccfd3974779ad5750ab4549c6ae P 132776399a2241e8b33be7693e69fd30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40a92445911542ddae31959ca2cbe467 in term 1.
I20260926 02:06:55.075040 20636 raft_consensus.cc:2454] T 6511bccfd3974779ad5750ab4549c6ae P dabdcdc9ec7a457a95e08b67b752e2cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40a92445911542ddae31959ca2cbe467 in term 1.
I20260926 02:06:55.075234 20422 leader_election.cc:302] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 132776399a2241e8b33be7693e69fd30, 40a92445911542ddae31959ca2cbe467; no voters:
I20260926 02:06:55.075796 20719 raft_consensus.cc:2781] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:06:55.075989 20719 raft_consensus.cc:686] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 [term 1 LEADER]: Becoming Leader. State: Replica: 40a92445911542ddae31959ca2cbe467, State: Running, Role: LEADER
I20260926 02:06:55.076406 20719 consensus_queue.cc:237] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } }
I20260926 02:06:55.079954 20195 catalog_manager.cc:5495] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40a92445911542ddae31959ca2cbe467 (127.18.159.130). New cstate: current_term: 1 leader_uuid: "40a92445911542ddae31959ca2cbe467" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } health_report { overall_health: UNKNOWN } } }
Sep 26 02:06:55 dist-test-slave-03rn krb5kdc[20150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388412, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:06:55.162010 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:55.143582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:45847 (local address 127.0.0.1:52780)
0926 02:06:55.144449 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:55.144472 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:55.144651 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:55.145112 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:55.145122 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:55.146113 (+ 991us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:06:55.146463 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:55.146476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.148028 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.148034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:55.149799 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:55.149818 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.150722 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.150729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:55.150959 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:55.151989 (+ 1030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:06:55.152034 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:06:55.154808 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:06:55.158832 (+ 4024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:55.158848 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:55.158857 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:55.159542 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:55.160002 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:55.160007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:55.160011 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:55.160238 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:55.160846 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:06:55.160855 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:06:55.161055 (+ 200us) client_negotiation.cc:769] Sending connection context
0926 02:06:55.161229 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 02:06:55.161577 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":372,"threads_started":1}
I20260926 02:06:55.162350 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:55.143566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.18.159.190:45847)
0926 02:06:55.144781 (+ 1215us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:55.144785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:55.144806 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:06:55.144854 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:55.144857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:55.144921 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:55.145017 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:55.146724 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.147539 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.149975 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.150458 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.150647 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:55.155520 (+ 4873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:55.155526 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:55.155530 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:55.155578 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:55.158649 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:55.159734 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:55.159739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:55.159741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:55.159886 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:55.160400 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:55.160404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:55.160406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:55.160698 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:55.160833 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:55.161381 (+ 548us) server_negotiation.cc:299] Negotiation successful
0926 02:06:55.161771 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":1097,"thread_start_us":951,"threads_started":1}
I20260926 02:06:55.168706 20195 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:52780:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 02:06:55.169764 20195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:06:55.177125 20332 tablet_service.cc:1467] Processing CreateTablet for tablet 89d4d383295c44179363b3246dfd8462 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=804dc6f7ddee4b90a5322dc83399d2f3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:06:55.177143 20472 tablet_service.cc:1467] Processing CreateTablet for tablet 89d4d383295c44179363b3246dfd8462 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=804dc6f7ddee4b90a5322dc83399d2f3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:06:55.177798 20472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89d4d383295c44179363b3246dfd8462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:06:55.177798 20332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89d4d383295c44179363b3246dfd8462. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 02:06:55.181998 20403 tablet.cc:2369] T 6511bccfd3974779ad5750ab4549c6ae P 132776399a2241e8b33be7693e69fd30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:06:55.185117 20611 tablet_service.cc:1467] Processing CreateTablet for tablet 89d4d383295c44179363b3246dfd8462 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=804dc6f7ddee4b90a5322dc83399d2f3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:06:55.185796 20611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89d4d383295c44179363b3246dfd8462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:06:55.187966 20702 tablet_bootstrap.cc:492] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467: Bootstrap starting.
I20260926 02:06:55.189886 20702 tablet_bootstrap.cc:654] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467: Neither blocks nor log segments found. Creating new log.
I20260926 02:06:55.198712 20700 tablet_bootstrap.cc:492] T 89d4d383295c44179363b3246dfd8462 P dabdcdc9ec7a457a95e08b67b752e2cd: Bootstrap starting.
I20260926 02:06:55.200376 20701 tablet_bootstrap.cc:492] T 89d4d383295c44179363b3246dfd8462 P 132776399a2241e8b33be7693e69fd30: Bootstrap starting.
I20260926 02:06:55.200857 20700 tablet_bootstrap.cc:654] T 89d4d383295c44179363b3246dfd8462 P dabdcdc9ec7a457a95e08b67b752e2cd: Neither blocks nor log segments found. Creating new log.
I20260926 02:06:55.202018 20702 tablet_bootstrap.cc:492] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467: No bootstrap required, opened a new log
I20260926 02:06:55.202250 20702 ts_tablet_manager.cc:1397] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20260926 02:06:55.202418 20701 tablet_bootstrap.cc:654] T 89d4d383295c44179363b3246dfd8462 P 132776399a2241e8b33be7693e69fd30: Neither blocks nor log segments found. Creating new log.
I20260926 02:06:55.203121 20702 raft_consensus.cc:348] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } }
I20260926 02:06:55.203433 20702 raft_consensus.cc:374] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:06:55.203560 20702 raft_consensus.cc:729] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a92445911542ddae31959ca2cbe467, State: Initialized, Role: FOLLOWER
I20260926 02:06:55.205646 20702 consensus_queue.cc:260] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } }
I20260926 02:06:55.206248 20719 tablet_replica.cc:439] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } } }
I20260926 02:06:55.206558 20719 tablet_replica.cc:442] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:06:55.209735 20702 ts_tablet_manager.cc:1428] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20260926 02:06:55.213248 20701 tablet_bootstrap.cc:492] T 89d4d383295c44179363b3246dfd8462 P 132776399a2241e8b33be7693e69fd30: No bootstrap required, opened a new log
I20260926 02:06:55.213510 20701 ts_tablet_manager.cc:1397] T 89d4d383295c44179363b3246dfd8462 P 132776399a2241e8b33be7693e69fd30: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260926 02:06:55.214423 20701 raft_consensus.cc:348] T 89d4d383295c44179363b3246dfd8462 P 132776399a2241e8b33be7693e69fd30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } }
I20260926 02:06:55.214709 20701 raft_consensus.cc:374] T 89d4d383295c44179363b3246dfd8462 P 132776399a2241e8b33be7693e69fd30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:06:55.214816 20701 raft_consensus.cc:729] T 89d4d383295c44179363b3246dfd8462 P 132776399a2241e8b33be7693e69fd30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 132776399a2241e8b33be7693e69fd30, State: Initialized, Role: FOLLOWER
I20260926 02:06:55.215116 20701 consensus_queue.cc:260] T 89d4d383295c44179363b3246dfd8462 P 132776399a2241e8b33be7693e69fd30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } }
I20260926 02:06:55.215817 20701 ts_tablet_manager.cc:1428] T 89d4d383295c44179363b3246dfd8462 P 132776399a2241e8b33be7693e69fd30: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 02:06:55.215636 20707 tablet_replica.cc:439] T 89d4d383295c44179363b3246dfd8462 P 132776399a2241e8b33be7693e69fd30: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } } }
I20260926 02:06:55.216068 20707 tablet_replica.cc:442] T 89d4d383295c44179363b3246dfd8462 P 132776399a2241e8b33be7693e69fd30: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:06:55.216246 20700 tablet_bootstrap.cc:492] T 89d4d383295c44179363b3246dfd8462 P dabdcdc9ec7a457a95e08b67b752e2cd: No bootstrap required, opened a new log
I20260926 02:06:55.216388 20700 ts_tablet_manager.cc:1397] T 89d4d383295c44179363b3246dfd8462 P dabdcdc9ec7a457a95e08b67b752e2cd: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
I20260926 02:06:55.217043 20700 raft_consensus.cc:348] T 89d4d383295c44179363b3246dfd8462 P dabdcdc9ec7a457a95e08b67b752e2cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } }
I20260926 02:06:55.217268 20700 raft_consensus.cc:374] T 89d4d383295c44179363b3246dfd8462 P dabdcdc9ec7a457a95e08b67b752e2cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:06:55.217348 20700 raft_consensus.cc:729] T 89d4d383295c44179363b3246dfd8462 P dabdcdc9ec7a457a95e08b67b752e2cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabdcdc9ec7a457a95e08b67b752e2cd, State: Initialized, Role: FOLLOWER
I20260926 02:06:55.217660 20700 consensus_queue.cc:260] T 89d4d383295c44179363b3246dfd8462 P dabdcdc9ec7a457a95e08b67b752e2cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } }
I20260926 02:06:55.218266 20700 ts_tablet_manager.cc:1428] T 89d4d383295c44179363b3246dfd8462 P dabdcdc9ec7a457a95e08b67b752e2cd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:06:55.218179 20706 tablet_replica.cc:439] T 89d4d383295c44179363b3246dfd8462 P dabdcdc9ec7a457a95e08b67b752e2cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } } }
I20260926 02:06:55.218508 20706 tablet_replica.cc:442] T 89d4d383295c44179363b3246dfd8462 P dabdcdc9ec7a457a95e08b67b752e2cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:06:55.297860 20719 raft_consensus.cc:482] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:06:55.298058 20719 raft_consensus.cc:504] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } }
I20260926 02:06:55.298863 20719 leader_election.cc:288] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 132776399a2241e8b33be7693e69fd30 (127.18.159.129:42417), dabdcdc9ec7a457a95e08b67b752e2cd (127.18.159.131:41629)
I20260926 02:06:55.299563 20356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d4d383295c44179363b3246dfd8462" candidate_uuid: "40a92445911542ddae31959ca2cbe467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "132776399a2241e8b33be7693e69fd30" is_pre_election: true
I20260926 02:06:55.299636 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d4d383295c44179363b3246dfd8462" candidate_uuid: "40a92445911542ddae31959ca2cbe467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" is_pre_election: true
I20260926 02:06:55.299861 20356 raft_consensus.cc:2454] T 89d4d383295c44179363b3246dfd8462 P 132776399a2241e8b33be7693e69fd30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40a92445911542ddae31959ca2cbe467 in term 0.
I20260926 02:06:55.299911 20636 raft_consensus.cc:2454] T 89d4d383295c44179363b3246dfd8462 P dabdcdc9ec7a457a95e08b67b752e2cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40a92445911542ddae31959ca2cbe467 in term 0.
I20260926 02:06:55.300436 20422 leader_election.cc:302] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 132776399a2241e8b33be7693e69fd30, 40a92445911542ddae31959ca2cbe467; no voters:
I20260926 02:06:55.300783 20719 raft_consensus.cc:2781] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:06:55.300897 20719 raft_consensus.cc:482] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:06:55.301076 20719 raft_consensus.cc:3037] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:06:55.303895 20719 raft_consensus.cc:504] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } }
I20260926 02:06:55.304694 20719 leader_election.cc:288] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467 [CANDIDATE]: Term 1 election: Requested vote from peers 132776399a2241e8b33be7693e69fd30 (127.18.159.129:42417), dabdcdc9ec7a457a95e08b67b752e2cd (127.18.159.131:41629)
I20260926 02:06:55.305267 20356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d4d383295c44179363b3246dfd8462" candidate_uuid: "40a92445911542ddae31959ca2cbe467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "132776399a2241e8b33be7693e69fd30"
I20260926 02:06:55.305423 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d4d383295c44179363b3246dfd8462" candidate_uuid: "40a92445911542ddae31959ca2cbe467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd"
I20260926 02:06:55.305560 20356 raft_consensus.cc:3037] T 89d4d383295c44179363b3246dfd8462 P 132776399a2241e8b33be7693e69fd30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:06:55.305693 20636 raft_consensus.cc:3037] T 89d4d383295c44179363b3246dfd8462 P dabdcdc9ec7a457a95e08b67b752e2cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:06:55.307727 20356 raft_consensus.cc:2454] T 89d4d383295c44179363b3246dfd8462 P 132776399a2241e8b33be7693e69fd30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40a92445911542ddae31959ca2cbe467 in term 1.
I20260926 02:06:55.308372 20422 leader_election.cc:302] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 132776399a2241e8b33be7693e69fd30, 40a92445911542ddae31959ca2cbe467; no voters:
I20260926 02:06:55.308740 20719 raft_consensus.cc:2781] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:06:55.308822 20636 raft_consensus.cc:2454] T 89d4d383295c44179363b3246dfd8462 P dabdcdc9ec7a457a95e08b67b752e2cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40a92445911542ddae31959ca2cbe467 in term 1.
I20260926 02:06:55.308967 20719 raft_consensus.cc:686] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467 [term 1 LEADER]: Becoming Leader. State: Replica: 40a92445911542ddae31959ca2cbe467, State: Running, Role: LEADER
I20260926 02:06:55.309252 20719 consensus_queue.cc:237] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } }
I20260926 02:06:55.311316 20711 tablet_replica.cc:439] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467: TxnStatusTablet state changed. Reason: New leader 40a92445911542ddae31959ca2cbe467. Latest consensus state: current_term: 1 leader_uuid: "40a92445911542ddae31959ca2cbe467" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } } }
I20260926 02:06:55.311673 20711 tablet_replica.cc:442] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:06:55.312465 20742 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 02:06:55.313331 20742 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 02:06:55.313707 20192 catalog_manager.cc:5495] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40a92445911542ddae31959ca2cbe467 (127.18.159.130). New cstate: current_term: 1 leader_uuid: "40a92445911542ddae31959ca2cbe467" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } health_report { overall_health: UNKNOWN } } }
I20260926 02:06:55.360361 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:55.353340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32784 (local address 127.18.159.130:46863)
0926 02:06:55.353919 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:55.353924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:55.353942 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:06:55.354002 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:55.354006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:55.354088 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:55.354198 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:06:55.354621 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.355464 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.357576 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.358299 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.358506 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:55.359103 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:06:55.359249 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:06:55.359325 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:55.359929 (+ 604us) server_negotiation.cc:299] Negotiation successful
0926 02:06:55.360008 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":325,"threads_started":1}
I20260926 02:06:55.360425 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:55.353286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:46863 (local address 127.0.0.1:32784)
0926 02:06:55.353554 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:55.353578 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:55.353747 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:55.354263 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:55.354268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:55.354283 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:06:55.354474 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:55.354484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.355619 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.355625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:55.357382 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:55.357403 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.358550 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.358557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:55.358783 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:55.358872 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:06:55.359355 (+ 483us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:06:55.359374 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 02:06:55.359463 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 02:06:55.360068 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 02:06:55.364239 20636 raft_consensus.cc:1261] T 89d4d383295c44179363b3246dfd8462 P dabdcdc9ec7a457a95e08b67b752e2cd [term 1 FOLLOWER]: Refusing update from remote peer 40a92445911542ddae31959ca2cbe467: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:06:55.364274 20356 raft_consensus.cc:1261] T 89d4d383295c44179363b3246dfd8462 P 132776399a2241e8b33be7693e69fd30 [term 1 FOLLOWER]: Refusing update from remote peer 40a92445911542ddae31959ca2cbe467: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:06:55.365064 20711 consensus_queue.cc:1035] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467 [LEADER]: Connected to new peer: Peer: permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:06:55.365466 20719 consensus_queue.cc:1035] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:06:55.374235 20706 tablet_replica.cc:439] T 89d4d383295c44179363b3246dfd8462 P dabdcdc9ec7a457a95e08b67b752e2cd: TxnStatusTablet state changed. Reason: New leader 40a92445911542ddae31959ca2cbe467. Latest consensus state: current_term: 1 leader_uuid: "40a92445911542ddae31959ca2cbe467" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } } }
I20260926 02:06:55.374527 20706 tablet_replica.cc:442] T 89d4d383295c44179363b3246dfd8462 P dabdcdc9ec7a457a95e08b67b752e2cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:06:55.376169 20745 mvcc.cc:205] Tried to move back new op lower bound from 7333430949325500416 to 7333430949109698560. Current Snapshot: MvccSnapshot[applied={T|T < 7333430949325500416}]
I20260926 02:06:55.376863 20719 tablet_replica.cc:439] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40a92445911542ddae31959ca2cbe467" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } } }
I20260926 02:06:55.377197 20719 tablet_replica.cc:442] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:06:55.380685 20707 tablet_replica.cc:439] T 89d4d383295c44179363b3246dfd8462 P 132776399a2241e8b33be7693e69fd30: TxnStatusTablet state changed. Reason: New leader 40a92445911542ddae31959ca2cbe467. Latest consensus state: current_term: 1 leader_uuid: "40a92445911542ddae31959ca2cbe467" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } } }
I20260926 02:06:55.380980 20707 tablet_replica.cc:442] T 89d4d383295c44179363b3246dfd8462 P 132776399a2241e8b33be7693e69fd30: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:06:55.382779 20711 tablet_replica.cc:439] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40a92445911542ddae31959ca2cbe467" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } } }
I20260926 02:06:55.383034 20711 tablet_replica.cc:442] T 89d4d383295c44179363b3246dfd8462 P 40a92445911542ddae31959ca2cbe467: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:06:55.385623 20706 tablet_replica.cc:439] T 89d4d383295c44179363b3246dfd8462 P dabdcdc9ec7a457a95e08b67b752e2cd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40a92445911542ddae31959ca2cbe467" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } } }
I20260926 02:06:55.385861 20706 tablet_replica.cc:442] T 89d4d383295c44179363b3246dfd8462 P dabdcdc9ec7a457a95e08b67b752e2cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:06:55.389489 20707 tablet_replica.cc:439] T 89d4d383295c44179363b3246dfd8462 P 132776399a2241e8b33be7693e69fd30: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40a92445911542ddae31959ca2cbe467" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 } } peers { permanent_uuid: "40a92445911542ddae31959ca2cbe467" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46863 } } peers { permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 } } }
I20260926 02:06:55.389762 20707 tablet_replica.cc:442] T 89d4d383295c44179363b3246dfd8462 P 132776399a2241e8b33be7693e69fd30: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:06:55.406189 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:55.399897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:36143 (local address 127.18.159.130:46863)
0926 02:06:55.400100 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:55.400109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:55.400829 (+ 720us) server_negotiation.cc:407] Connection header received
0926 02:06:55.401004 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:55.401010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:55.401085 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:55.401228 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:06:55.401823 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.402797 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.404330 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.404916 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.405036 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:55.405451 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:06:55.405575 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:06:55.405660 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:55.405793 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 02:06:55.405866 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:06:55.406212 20755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:55.399724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:46863 (local address 127.18.159.130:36143)
0926 02:06:55.400707 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:55.400727 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:55.400882 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:55.401242 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:55.401247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:55.401259 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:06:55.401663 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:55.401676 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.402925 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.402929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:55.404136 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:55.404148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.405104 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.405109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:55.405237 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:55.405295 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:06:55.405675 (+ 380us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:06:55.405684 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 02:06:55.405753 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 02:06:55.405857 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":599,"threads_started":1}
I20260926 02:06:55.415465 20356 raft_consensus.cc:1261] T 6511bccfd3974779ad5750ab4549c6ae P 132776399a2241e8b33be7693e69fd30 [term 1 FOLLOWER]: Refusing update from remote peer 40a92445911542ddae31959ca2cbe467: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:06:55.415503 20636 raft_consensus.cc:1261] T 6511bccfd3974779ad5750ab4549c6ae P dabdcdc9ec7a457a95e08b67b752e2cd [term 1 FOLLOWER]: Refusing update from remote peer 40a92445911542ddae31959ca2cbe467: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:06:55.416038 20711 consensus_queue.cc:1035] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 [LEADER]: Connected to new peer: Peer: permanent_uuid: "132776399a2241e8b33be7693e69fd30" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:06:55.416404 20719 consensus_queue.cc:1035] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdcdc9ec7a457a95e08b67b752e2cd" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:06:55.487171 20195 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:06:55.487902 20195 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:06:55.491225 20195 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bab9c0d8559c4bbda907f8225a486133: Sending DeleteTablet for 3 replicas of tablet 6511bccfd3974779ad5750ab4549c6ae
I20260926 02:06:55.492063 20332 tablet_service.cc:1514] Processing DeleteTablet for tablet 6511bccfd3974779ad5750ab4549c6ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:06:55 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:54208
I20260926 02:06:55.492167 20472 tablet_service.cc:1514] Processing DeleteTablet for tablet 6511bccfd3974779ad5750ab4549c6ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:06:55 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:60986
I20260926 02:06:55.492331 20611 tablet_service.cc:1514] Processing DeleteTablet for tablet 6511bccfd3974779ad5750ab4549c6ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:06:55 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:47978
I20260926 02:06:55.492739 20761 tablet_replica.cc:330] T 6511bccfd3974779ad5750ab4549c6ae P 132776399a2241e8b33be7693e69fd30: stopping tablet replica
I20260926 02:06:55.492970 20761 raft_consensus.cc:2229] T 6511bccfd3974779ad5750ab4549c6ae P 132776399a2241e8b33be7693e69fd30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:06:55.493242 20761 raft_consensus.cc:2258] T 6511bccfd3974779ad5750ab4549c6ae P 132776399a2241e8b33be7693e69fd30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:06:55.494377 20761 ts_tablet_manager.cc:1905] T 6511bccfd3974779ad5750ab4549c6ae P 132776399a2241e8b33be7693e69fd30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:06:55.502375 20761 ts_tablet_manager.cc:1918] T 6511bccfd3974779ad5750ab4549c6ae P 132776399a2241e8b33be7693e69fd30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:06:55.502928 20763 tablet_replica.cc:330] T 6511bccfd3974779ad5750ab4549c6ae P dabdcdc9ec7a457a95e08b67b752e2cd: stopping tablet replica
I20260926 02:06:55.503233 20763 raft_consensus.cc:2229] T 6511bccfd3974779ad5750ab4549c6ae P dabdcdc9ec7a457a95e08b67b752e2cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:06:55.503538 20763 raft_consensus.cc:2258] T 6511bccfd3974779ad5750ab4549c6ae P dabdcdc9ec7a457a95e08b67b752e2cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:06:55.505184 20763 ts_tablet_manager.cc:1905] T 6511bccfd3974779ad5750ab4549c6ae P dabdcdc9ec7a457a95e08b67b752e2cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:06:55.505687 20762 tablet_replica.cc:330] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467: stopping tablet replica
I20260926 02:06:55.505910 20762 raft_consensus.cc:2229] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:06:55.506296 20762 raft_consensus.cc:2258] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:06:55.507314 20762 ts_tablet_manager.cc:1905] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:06:55.513278 20761 log.cc:1196] T 6511bccfd3974779ad5750ab4549c6ae P 132776399a2241e8b33be7693e69fd30: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/6511bccfd3974779ad5750ab4549c6ae
I20260926 02:06:55.514002 20761 ts_tablet_manager.cc:1939] T 6511bccfd3974779ad5750ab4549c6ae P 132776399a2241e8b33be7693e69fd30: Deleting consensus metadata
I20260926 02:06:55.515486 20179 catalog_manager.cc:4841] TS 132776399a2241e8b33be7693e69fd30 (127.18.159.129:42417): tablet 6511bccfd3974779ad5750ab4549c6ae (table test-table [id=16dbf1dbf71e4e02a3a603c08c145f2b]) successfully deleted
I20260926 02:06:55.516072 20762 ts_tablet_manager.cc:1918] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:06:55.516232 20762 log.cc:1196] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/6511bccfd3974779ad5750ab4549c6ae
I20260926 02:06:55.516762 20762 ts_tablet_manager.cc:1939] T 6511bccfd3974779ad5750ab4549c6ae P 40a92445911542ddae31959ca2cbe467: Deleting consensus metadata
I20260926 02:06:55.517696 20763 ts_tablet_manager.cc:1918] T 6511bccfd3974779ad5750ab4549c6ae P dabdcdc9ec7a457a95e08b67b752e2cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:06:55.517887 20763 log.cc:1196] T 6511bccfd3974779ad5750ab4549c6ae P dabdcdc9ec7a457a95e08b67b752e2cd: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/6511bccfd3974779ad5750ab4549c6ae
I20260926 02:06:55.517866 20182 catalog_manager.cc:4841] TS 40a92445911542ddae31959ca2cbe467 (127.18.159.130:46863): tablet 6511bccfd3974779ad5750ab4549c6ae (table test-table [id=16dbf1dbf71e4e02a3a603c08c145f2b]) successfully deleted
I20260926 02:06:55.518458 20763 ts_tablet_manager.cc:1939] T 6511bccfd3974779ad5750ab4549c6ae P dabdcdc9ec7a457a95e08b67b752e2cd: Deleting consensus metadata
I20260926 02:06:55.519706 20179 catalog_manager.cc:4841] TS dabdcdc9ec7a457a95e08b67b752e2cd (127.18.159.131:41629): tablet 6511bccfd3974779ad5750ab4549c6ae (table test-table [id=16dbf1dbf71e4e02a3a603c08c145f2b]) successfully deleted
Sep 26 02:06:55 dist-test-slave-03rn krb5kdc[20150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388414, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:06:55.542532 20765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:55.505795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54224 (local address 127.18.159.129:42417)
0926 02:06:55.513702 (+ 7907us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:55.513709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:55.524013 (+ 10304us) server_negotiation.cc:407] Connection header received
0926 02:06:55.524188 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:55.524194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:55.524288 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:55.524425 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:55.526406 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.527658 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.529070 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.529780 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.529922 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:55.534792 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:55.534814 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:55.534835 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:55.534895 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:55.538862 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:55.539781 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:55.539795 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:55.539801 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:55.539988 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:55.540575 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:55.540581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:55.540585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:55.541053 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:55.541362 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:55.541824 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 02:06:55.542079 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7682,"thread_start_us":7537,"threads_started":1}
W20260926 02:06:55.543454 20397 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54224
I20260926 02:06:55.564905 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:55.546713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52796 (local address 127.18.159.190:45847)
0926 02:06:55.546968 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:55.546975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:55.547160 (+ 185us) server_negotiation.cc:407] Connection header received
0926 02:06:55.547336 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:55.547340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:55.547402 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:55.547485 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:55.549273 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.550429 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.553063 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.553791 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.553933 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:55.557644 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:55.557651 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:55.557657 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:55.557706 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:55.561816 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:55.562785 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:55.562790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:55.562795 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:55.562944 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:55.563357 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:55.563362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:55.563366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:55.563715 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:55.563929 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:55.564359 (+ 430us) server_negotiation.cc:299] Negotiation successful
0926 02:06:55.564584 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
W20260926 02:06:55.565407 20195 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52796
I20260926 02:06:55.567245 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 20266
I20260926 02:06:55.588801 20765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:55.568573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.18.159.129:42417)
0926 02:06:55.568770 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:55.568776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:55.572729 (+ 3953us) server_negotiation.cc:407] Connection header received
0926 02:06:55.572868 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:55.572874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:55.572961 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:55.573075 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:55.574593 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.575702 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.577635 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.578330 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.578474 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:55.581954 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:55.581962 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:55.581967 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:55.582018 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:55.585794 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:55.586650 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:55.586656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:55.586660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:55.586824 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:55.587222 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:55.587227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:55.587231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:55.587607 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:55.587847 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:55.588144 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 02:06:55.588429 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 02:06:55.589483 20397 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54228
W20260926 02:06:55.590096 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 20266: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:06:55.590358 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 20266
I20260926 02:06:55.613602 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 20406
Sep 26 02:06:55 dist-test-slave-03rn krb5kdc[20150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388414, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.159.130@KRBTEST.COM
I20260926 02:06:55.632102 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:55.614514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32794 (local address 127.18.159.130:46863)
0926 02:06:55.614707 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:55.614714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:55.614766 (+ 52us) server_negotiation.cc:407] Connection header received
0926 02:06:55.614934 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:55.614940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:55.615031 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:55.615144 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:55.616462 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.617665 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.619180 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.619949 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.620085 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:55.624576 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:55.624597 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:55.624618 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:55.624667 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:55.628858 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:55.629651 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:55.629662 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:55.629666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:55.629853 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:55.630281 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:55.630287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:55.630290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:55.630798 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:55.631119 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:55.631430 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 02:06:55.631693 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 02:06:55.633256 20537 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32794
W20260926 02:06:55.634150 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 20406: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:06:55.634330 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 20406
I20260926 02:06:55.657528 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 20546
Sep 26 02:06:55 dist-test-slave-03rn krb5kdc[20150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388414, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.159.131@KRBTEST.COM
I20260926 02:06:55.674849 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:55.658250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47982 (local address 127.18.159.131:41629)
0926 02:06:55.658723 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:55.658728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:55.658745 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:06:55.658796 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:55.658799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:55.658856 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:55.658973 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:55.660425 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.661222 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.662612 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:55.663031 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:55.663146 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:55.667971 (+ 4825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:55.667988 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:55.668006 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:55.668054 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:55.671389 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:55.672509 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:55.672518 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:55.672520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:55.672700 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:55.673268 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:55.673272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:55.673274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:55.673697 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:55.673947 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:55.674276 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 02:06:55.674458 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":250,"threads_started":1}
W20260926 02:06:55.675638 20676 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47982
W20260926 02:06:55.676465 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 20546: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:06:55.676669 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 20546
I20260926 02:06:55.693516 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 20163
I20260926 02:06:55.694727 20235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:06:55.809664 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 20163
2026-09-26T02:06:55Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5130 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:06:55 dist-test-slave-03rn krb5kdc[20774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:06:55 dist-test-slave-03rn krb5kdc[20774](info): set up 2 sockets
Sep 26 02:06:55 dist-test-slave-03rn krb5kdc[20774](info): commencing operation
krb5kdc: starting...
W20260926 02:06:56.900923 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:06:57 dist-test-slave-03rn krb5kdc[20774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:06:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:06:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:06:57.103581 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:39677
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:39677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:06:57.443202 20787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:06:57.443558 20787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:06:57.443701 20787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:06:57.451896 20787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:06:57.452023 20787 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 02:06:57.452108 20787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:06:57.452235 20787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:06:57.452308 20787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:06:57.452409 20787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:06:57.462716 20787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:39677
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:39677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:06:57.464087 20787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:06:57.465533 20787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:06:57.478272 20792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:57.478430 20793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:57.478430 20795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:06:57.479844 20787 server_base.cc:1034] running on GCE node
I20260926 02:06:57.480854 20787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:06:57.482522 20787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:06:57.483812 20787 hybrid_clock.cc:648] HybridClock initialized: now 1790388417483727 us; error 67 us; skew 500 ppm
I20260926 02:06:57.484306 20787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:06:57 dist-test-slave-03rn krb5kdc[20774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388417, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:06:57.487843 20787 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:06:57.489656 20787 webserver.cc:460] Webserver started at http://127.18.159.190:42659/ using document root <none> and password file <none>
I20260926 02:06:57.490439 20787 fs_manager.cc:359] Metadata directory not provided
I20260926 02:06:57.490581 20787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:06:57.490908 20787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:06:57.494084 20787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "e76c06d8fc9142adafcd660c2102fa24"
format_stamp: "Formatted at 2026-09-26 02:06:57 on dist-test-slave-03rn"
server_key: "91eeb35c8611aef11a136e52dc27e3bb"
server_key_iv: "448eda181fa36ce028f7e4f98b86e3b0"
server_key_version: "encryptionkey@0"
I20260926 02:06:57.494899 20787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "e76c06d8fc9142adafcd660c2102fa24"
format_stamp: "Formatted at 2026-09-26 02:06:57 on dist-test-slave-03rn"
server_key: "91eeb35c8611aef11a136e52dc27e3bb"
server_key_iv: "448eda181fa36ce028f7e4f98b86e3b0"
server_key_version: "encryptionkey@0"
I20260926 02:06:57.500598 20787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 02:06:57.505127 20802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:57.506668 20787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:06:57.506898 20787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "e76c06d8fc9142adafcd660c2102fa24"
format_stamp: "Formatted at 2026-09-26 02:06:57 on dist-test-slave-03rn"
server_key: "91eeb35c8611aef11a136e52dc27e3bb"
server_key_iv: "448eda181fa36ce028f7e4f98b86e3b0"
server_key_version: "encryptionkey@0"
I20260926 02:06:57.507160 20787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:06:57.522287 20787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:06:57.526831 20787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:06:57.527122 20787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:06:57.549396 20787 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:39677
I20260926 02:06:57.549402 20863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:39677 every 8 connection(s)
I20260926 02:06:57.550890 20787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:06:57.553730 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 20787
I20260926 02:06:57.553984 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:06:57.554790 19070 external_mini_cluster.cc:1428] Setting key bbc49976ac3b84db30394478f60dc991
I20260926 02:06:57.555949 20864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:06:57 dist-test-slave-03rn krb5kdc[20774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:06:57.569444 20864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24: Bootstrap starting.
I20260926 02:06:57.573297 20864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24: Neither blocks nor log segments found. Creating new log.
I20260926 02:06:57.574447 20864 log.cc:824] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24: Log is configured to *not* fsync() on all Append() calls
I20260926 02:06:57.577646 20864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24: No bootstrap required, opened a new log
I20260926 02:06:57.577791 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:57.556982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60004 (local address 127.18.159.190:39677)
0926 02:06:57.557785 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:57.557798 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:57.557839 (+ 41us) server_negotiation.cc:407] Connection header received
0926 02:06:57.559392 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:57.559416 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:57.559888 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:57.560395 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:57.561870 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:57.563263 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:57.564675 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:57.565378 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:57.565614 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:57.570036 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:57.570044 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:57.570062 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:57.570106 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:57.573329 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:57.574160 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:57.574172 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:57.574180 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:57.574354 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:57.574832 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:57.574836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:57.574838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:57.575430 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:57.575774 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:57.576306 (+ 532us) server_negotiation.cc:299] Negotiation successful
0926 02:06:57.576735 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":365,"threads_started":1}
I20260926 02:06:57.586756 20864 raft_consensus.cc:348] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76c06d8fc9142adafcd660c2102fa24" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39677 } }
I20260926 02:06:57.587216 20864 raft_consensus.cc:374] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:06:57.587332 20864 raft_consensus.cc:729] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e76c06d8fc9142adafcd660c2102fa24, State: Initialized, Role: FOLLOWER
I20260926 02:06:57.588002 20864 consensus_queue.cc:260] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76c06d8fc9142adafcd660c2102fa24" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39677 } }
I20260926 02:06:57.588271 20864 raft_consensus.cc:388] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:06:57.588418 20864 raft_consensus.cc:482] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:06:57.588599 20864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:06:57.590837 20864 raft_consensus.cc:504] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76c06d8fc9142adafcd660c2102fa24" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39677 } }
I20260926 02:06:57.591517 20864 leader_election.cc:302] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e76c06d8fc9142adafcd660c2102fa24; no voters:
I20260926 02:06:57.592307 20864 leader_election.cc:288] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:06:57.592429 20869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:06:57.593233 20869 raft_consensus.cc:686] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24 [term 1 LEADER]: Becoming Leader. State: Replica: e76c06d8fc9142adafcd660c2102fa24, State: Running, Role: LEADER
I20260926 02:06:57.593835 20869 consensus_queue.cc:237] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76c06d8fc9142adafcd660c2102fa24" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39677 } }
I20260926 02:06:57.594683 20864 sys_catalog.cc:564] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:06:57.597060 20871 sys_catalog.cc:455] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e76c06d8fc9142adafcd660c2102fa24. Latest consensus state: current_term: 1 leader_uuid: "e76c06d8fc9142adafcd660c2102fa24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76c06d8fc9142adafcd660c2102fa24" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39677 } } }
I20260926 02:06:57.597440 20871 sys_catalog.cc:458] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24 [sys.catalog]: This master's current role is: LEADER
I20260926 02:06:57.597198 20870 sys_catalog.cc:455] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e76c06d8fc9142adafcd660c2102fa24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76c06d8fc9142adafcd660c2102fa24" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39677 } } }
I20260926 02:06:57.597570 20870 sys_catalog.cc:458] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24 [sys.catalog]: This master's current role is: LEADER
I20260926 02:06:57.598459 20876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:06:57.605407 20876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:06:57.615717 20876 catalog_manager.cc:1347] Generated new cluster ID: f278a8494ebd4bd48adb51dd4c691524
I20260926 02:06:57.615859 20876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:06:57.631837 20876 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:06:57.633599 20876 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:06:57.647006 20876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24: Generated new TSK 0
I20260926 02:06:57.648322 20876 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.159.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
I20260926 02:06:57.711699 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:39677
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:06:58.080924 20890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:06:58.081318 20890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:06:58.081501 20890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:06:58.090489 20890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:06:58.090703 20890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:06:58.090852 20890 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 02:06:58.090942 20890 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:06:58.091085 20890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:06:58.102769 20890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.159.190:39677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:06:58.104264 20890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:06:58.105716 20890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:06:58.121595 20895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:58.121592 20896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:58.121713 20898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:06:58.122848 20890 server_base.cc:1034] running on GCE node
I20260926 02:06:58.123679 20890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:06:58.124845 20890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:06:58.126106 20890 hybrid_clock.cc:648] HybridClock initialized: now 1790388418126049 us; error 55 us; skew 500 ppm
I20260926 02:06:58.126461 20890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:06:58 dist-test-slave-03rn krb5kdc[20774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388418, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:06:58.135298 20890 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:06:58.137326 20890 webserver.cc:460] Webserver started at http://127.18.159.129:36239/ using document root <none> and password file <none>
I20260926 02:06:58.138243 20890 fs_manager.cc:359] Metadata directory not provided
I20260926 02:06:58.138602 20890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:06:58.139034 20890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:06:58.142380 20890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43"
format_stamp: "Formatted at 2026-09-26 02:06:58 on dist-test-slave-03rn"
server_key: "45a415f4ebcdac046b0fc87e62c88b5d"
server_key_iv: "d16ccbe359308546d642f4d3bf84bd81"
server_key_version: "encryptionkey@0"
I20260926 02:06:58.143266 20890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43"
format_stamp: "Formatted at 2026-09-26 02:06:58 on dist-test-slave-03rn"
server_key: "45a415f4ebcdac046b0fc87e62c88b5d"
server_key_iv: "d16ccbe359308546d642f4d3bf84bd81"
server_key_version: "encryptionkey@0"
I20260926 02:06:58.150166 20890 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260926 02:06:58.154819 20905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:58.156724 20890 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:06:58.156960 20890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43"
format_stamp: "Formatted at 2026-09-26 02:06:58 on dist-test-slave-03rn"
server_key: "45a415f4ebcdac046b0fc87e62c88b5d"
server_key_iv: "d16ccbe359308546d642f4d3bf84bd81"
server_key_version: "encryptionkey@0"
I20260926 02:06:58.157250 20890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:06:58.175995 20890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:06:58.180939 20890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:06:58.181322 20890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:06:58.184401 20890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:06:58.184542 20890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:58.184686 20890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:06:58.184790 20890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:06:58 dist-test-slave-03rn krb5kdc[20774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388418, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:06:58.212472 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:58.184958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:44535 (local address 127.18.159.190:39677)
0926 02:06:58.185866 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:58.185872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:58.187109 (+ 1237us) server_negotiation.cc:407] Connection header received
0926 02:06:58.189543 (+ 2434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:58.189549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:58.189663 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:58.189800 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:58.193833 (+ 4033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:58.194988 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:58.197749 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:58.198413 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:58.198573 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:58.203975 (+ 5402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:58.203982 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:58.203985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:58.204059 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:58.207828 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:58.209161 (+ 1333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:58.209167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:58.209171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:58.209369 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:58.210185 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:58.210188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:58.210190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:58.210463 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:58.210584 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:58.211801 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0926 02:06:58.212069 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":624,"threads_started":1}
I20260926 02:06:58.214589 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:58.185880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:39677 (local address 127.18.159.129:44535)
0926 02:06:58.186887 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:58.186936 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:58.189173 (+ 2237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:58.190329 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:58.190346 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:58.191727 (+ 1381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:06:58.193542 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:58.193593 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:58.195213 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:58.195217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:58.197508 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:58.197537 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:58.198708 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:58.198716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:58.198994 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:58.199788 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:06:58.199812 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:06:58.203642 (+ 3830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:06:58.208031 (+ 4389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:58.208042 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:58.208066 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:58.208970 (+ 904us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:58.209555 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:58.209565 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:58.209571 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:58.209982 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:58.210608 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:06:58.210621 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:06:58.211304 (+ 683us) client_negotiation.cc:769] Sending connection context
0926 02:06:58.211702 (+ 398us) client_negotiation.cc:240] Negotiation successful
0926 02:06:58.212546 (+ 844us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":441,"threads_started":1}
I20260926 02:06:58.241700 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:58.223374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:59459 (local address 127.18.159.190:39677)
0926 02:06:58.223605 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:58.223615 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:58.224920 (+ 1305us) server_negotiation.cc:407] Connection header received
0926 02:06:58.225072 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:58.225081 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:58.225210 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:58.225360 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:58.226370 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:58.227876 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:58.230238 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:58.230997 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:58.231169 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:58.234751 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:58.234761 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:58.234766 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:58.234819 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:58.238356 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:58.239525 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:58.239536 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:58.239538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:58.239687 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:58.240122 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:58.240132 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:58.240134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:58.240489 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:58.240672 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:58.241092 (+ 420us) server_negotiation.cc:299] Negotiation successful
0926 02:06:58.241334 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:06:58.241868 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:58.222856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:39677 (local address 127.18.159.129:59459)
0926 02:06:58.224757 (+ 1901us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:58.224780 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:58.224949 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:58.225408 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:58.225412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:58.225870 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:06:58.226194 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:58.226205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:58.228088 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:58.228093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:58.230031 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:58.230051 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:58.231563 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:58.231570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:58.231792 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:58.232868 (+ 1076us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:06:58.232887 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:06:58.234416 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:06:58.238613 (+ 4197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:58.238619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:58.238623 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:58.239355 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:58.239836 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:58.239841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:58.239845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:58.239986 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:58.240627 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:06:58.240633 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:06:58.240832 (+ 199us) client_negotiation.cc:769] Sending connection context
0926 02:06:58.240973 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 02:06:58.241392 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1729,"thread_start_us":1579,"threads_started":1}
I20260926 02:06:58.245442 20890 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:34257
I20260926 02:06:58.245592 21025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:34257 every 8 connection(s)
I20260926 02:06:58.247184 20890 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:06:58.249560 21026 heartbeater.cc:347] Connected to a master server at 127.18.159.190:39677
I20260926 02:06:58.249812 21026 heartbeater.cc:464] Registering TS with master...
I20260926 02:06:58.250545 21026 heartbeater.cc:511] Master 127.18.159.190:39677 requested a full tablet report, sending...
I20260926 02:06:58.252805 20819 ts_manager.cc:194] Registered new tserver with Master: 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 (127.18.159.129:34257)
I20260926 02:06:58.254400 20819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:44535
I20260926 02:06:58.255581 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 20890
I20260926 02:06:58.255854 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:06:58.256747 19070 external_mini_cluster.cc:1428] Setting key 6f8e3fdec1e7862e4125e25448e2a177
WARNING: no policy specified for kudu/127.18.159.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
I20260926 02:06:58.303470 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:39677
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:06:58.681671 21030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:06:58.682111 21030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:06:58.682319 21030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:06:58.691154 21030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:06:58.691401 21030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:06:58.691555 21030 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 02:06:58.691642 21030 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:06:58.691789 21030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:06:58.704566 21030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.159.190:39677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:06:58.706542 21030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:06:58.708400 21030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:06:58.726379 21035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:58.726490 21038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:06:58.726413 21036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:06:58.728348 21030 server_base.cc:1034] running on GCE node
I20260926 02:06:58.729290 21030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:06:58.730401 21030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:06:58.731703 21030 hybrid_clock.cc:648] HybridClock initialized: now 1790388418731661 us; error 75 us; skew 500 ppm
I20260926 02:06:58.732107 21030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:06:58 dist-test-slave-03rn krb5kdc[20774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388418, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:06:58.736383 21030 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:06:58.738472 21030 webserver.cc:460] Webserver started at http://127.18.159.130:40245/ using document root <none> and password file <none>
I20260926 02:06:58.739431 21030 fs_manager.cc:359] Metadata directory not provided
I20260926 02:06:58.739629 21030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:06:58.740022 21030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:06:58.743551 21030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "dd710cc670f54fa491a96609dcb96ce9"
format_stamp: "Formatted at 2026-09-26 02:06:58 on dist-test-slave-03rn"
server_key: "f50e8be864d398d3042d51442e345e33"
server_key_iv: "061c832b821ab754b9e3d4c536ff2326"
server_key_version: "encryptionkey@0"
I20260926 02:06:58.744736 21030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "dd710cc670f54fa491a96609dcb96ce9"
format_stamp: "Formatted at 2026-09-26 02:06:58 on dist-test-slave-03rn"
server_key: "f50e8be864d398d3042d51442e345e33"
server_key_iv: "061c832b821ab754b9e3d4c536ff2326"
server_key_version: "encryptionkey@0"
I20260926 02:06:58.751772 21030 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260926 02:06:58.756759 21045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:58.758654 21030 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:06:58.758905 21030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "dd710cc670f54fa491a96609dcb96ce9"
format_stamp: "Formatted at 2026-09-26 02:06:58 on dist-test-slave-03rn"
server_key: "f50e8be864d398d3042d51442e345e33"
server_key_iv: "061c832b821ab754b9e3d4c536ff2326"
server_key_version: "encryptionkey@0"
I20260926 02:06:58.759177 21030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:06:58.789346 21030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:06:58.794366 21030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:06:58.794780 21030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:06:58.803115 21030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:06:58.803282 21030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:06:58.803426 21030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:06:58.803532 21030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:06:58 dist-test-slave-03rn krb5kdc[20774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388418, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:06:58.830140 21053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:58.799730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:55517 (local address 127.18.159.190:39677)
0926 02:06:58.801921 (+ 2191us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:58.801927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:58.802965 (+ 1038us) server_negotiation.cc:407] Connection header received
0926 02:06:58.806382 (+ 3417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:58.806388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:58.806490 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:58.806610 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:58.809728 (+ 3118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:58.811330 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:58.815020 (+ 3690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:58.815870 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:58.816052 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:58.821255 (+ 5203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:58.821337 (+ 82us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:58.821343 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:58.821404 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:58.825446 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:58.827056 (+ 1610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:58.827061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:58.827063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:58.827299 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:58.828022 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:58.828027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:58.828029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:58.828395 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:58.828599 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:58.829467 (+ 868us) server_negotiation.cc:299] Negotiation successful
0926 02:06:58.829737 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1909,"thread_start_us":1740,"threads_started":1}
I20260926 02:06:58.831593 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:58.800557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:39677 (local address 127.18.159.130:55517)
0926 02:06:58.802656 (+ 2099us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:58.802719 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:58.805921 (+ 3202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:58.807050 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:58.807065 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:58.808075 (+ 1010us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:06:58.809314 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:58.809368 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:58.811953 (+ 2585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:58.811967 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:58.814719 (+ 2752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:58.814776 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:58.816202 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:58.816211 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:58.816520 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:58.817370 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:06:58.817424 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:06:58.820905 (+ 3481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:06:58.825731 (+ 4826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:58.825750 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:58.825779 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:58.826805 (+ 1026us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:58.827446 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:58.827454 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:58.827460 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:58.827843 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:58.828578 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:06:58.828589 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:06:58.829023 (+ 434us) client_negotiation.cc:769] Sending connection context
0926 02:06:58.829338 (+ 315us) client_negotiation.cc:240] Negotiation successful
0926 02:06:58.829988 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1545,"thread_start_us":1202,"threads_started":1}
I20260926 02:06:58.856104 21053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:58.838532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:60021 (local address 127.18.159.190:39677)
0926 02:06:58.838817 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 02:06:58.838824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:06:58.840677 (+ 1853us) server_negotiation.cc:407] Connection header received
0926 02:06:58.840931 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:06:58.840940 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:06:58.841015 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:06:58.841128 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:06:58.842335 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:58.843144 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:58.844800 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:06:58.845260 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:06:58.845421 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:58.849151 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:06:58.849160 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:06:58.849165 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:06:58.849214 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:06:58.852974 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:58.853965 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:58.853972 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:58.853975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:58.854113 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:06:58.854523 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:06:58.854528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:06:58.854533 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:06:58.854888 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:06:58.855070 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 02:06:58.855487 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 02:06:58.855741 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260926 02:06:58.856483 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:06:58.839305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:39677 (local address 127.18.159.130:60021)
0926 02:06:58.840288 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0926 02:06:58.840313 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:06:58.840773 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:06:58.841168 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:06:58.841173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:06:58.841729 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:06:58.842126 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:58.842156 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:58.843328 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:58.843335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:58.844611 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:06:58.844628 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:06:58.845501 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:06:58.845508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:06:58.845788 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:06:58.846791 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:06:58.846862 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:06:58.848707 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:06:58.853153 (+ 4446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:58.853159 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:58.853163 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:58.853804 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:58.854245 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:06:58.854250 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:06:58.854254 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:06:58.854399 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:06:58.855026 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:06:58.855031 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:06:58.855226 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 02:06:58.855371 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 02:06:58.855765 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"thread_start_us":581,"threads_started":1}
I20260926 02:06:58.878270 21165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:38649 every 8 connection(s)
I20260926 02:06:58.878014 21030 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:38649
I20260926 02:06:58.879987 21030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:06:58.882918 21166 heartbeater.cc:347] Connected to a master server at 127.18.159.190:39677
I20260926 02:06:58.883155 21166 heartbeater.cc:464] Registering TS with master...
I20260926 02:06:58.884292 21166 heartbeater.cc:511] Master 127.18.159.190:39677 requested a full tablet report, sending...
I20260926 02:06:58.885742 20819 ts_manager.cc:194] Registered new tserver with Master: dd710cc670f54fa491a96609dcb96ce9 (127.18.159.130:38649)
I20260926 02:06:58.886610 20819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:55517
I20260926 02:06:58.888083 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 21030
I20260926 02:06:58.888319 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:06:58.888743 19070 external_mini_cluster.cc:1428] Setting key df24a1c24ef9b2f92e077b6e041e7419
WARNING: no policy specified for kudu/127.18.159.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.131@KRBTEST.COM" created.
I20260926 02:06:59.258795 21026 heartbeater.cc:503] Master 127.18.159.190:39677 was elected leader, sending a full tablet report...
I20260926 02:06:59.889505 21166 heartbeater.cc:503] Master 127.18.159.190:39677 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
W20260926 02:07:00.194700 19070 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.18.159.131: real 1.304s user 0.000s sys 0.009s
I20260926 02:07:00.195520 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:39677
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:00.568543 21170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:00.568898 21170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:00.569073 21170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:00.577095 21170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:00.577286 21170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:00.577445 21170 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 02:07:00.577529 21170 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:07:00.577641 21170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:07:00.588932 21170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.159.190:39677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:00.590404 21170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:00.591988 21170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:00.607398 21175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:00.607333 21176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:00.607353 21178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:00.613538 21170 server_base.cc:1034] running on GCE node
I20260926 02:07:00.614372 21170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:00.615480 21170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:00.616722 21170 hybrid_clock.cc:648] HybridClock initialized: now 1790388420616662 us; error 55 us; skew 500 ppm
I20260926 02:07:00.617092 21170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:00 dist-test-slave-03rn krb5kdc[20774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388420, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:00.620793 21170 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:07:00.622608 21170 webserver.cc:460] Webserver started at http://127.18.159.131:44005/ using document root <none> and password file <none>
I20260926 02:07:00.623369 21170 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:00.623524 21170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:00.623868 21170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:00.627460 21170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "441d63ac043b43ce9e79d6a7f53c5434"
format_stamp: "Formatted at 2026-09-26 02:07:00 on dist-test-slave-03rn"
server_key: "3bdd142b3e47210dc1cb8f04f5916e8f"
server_key_iv: "44f3b05a8efe4981e2979b13401d4c75"
server_key_version: "encryptionkey@0"
I20260926 02:07:00.628371 21170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "441d63ac043b43ce9e79d6a7f53c5434"
format_stamp: "Formatted at 2026-09-26 02:07:00 on dist-test-slave-03rn"
server_key: "3bdd142b3e47210dc1cb8f04f5916e8f"
server_key_iv: "44f3b05a8efe4981e2979b13401d4c75"
server_key_version: "encryptionkey@0"
I20260926 02:07:00.634611 21170 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260926 02:07:00.639204 21185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:00.640960 21170 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 02:07:00.641158 21170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "441d63ac043b43ce9e79d6a7f53c5434"
format_stamp: "Formatted at 2026-09-26 02:07:00 on dist-test-slave-03rn"
server_key: "3bdd142b3e47210dc1cb8f04f5916e8f"
server_key_iv: "44f3b05a8efe4981e2979b13401d4c75"
server_key_version: "encryptionkey@0"
I20260926 02:07:00.641394 21170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:00.670524 21170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:00.675896 21170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:00.676332 21170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:00.679365 21170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:00.679509 21170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:00.679646 21170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:00.679730 21170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:07:00 dist-test-slave-03rn krb5kdc[20774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388420, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:00.707299 21194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:00.680415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:59611 (local address 127.18.159.190:39677)
0926 02:07:00.680833 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:00.680838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:00.682746 (+ 1908us) server_negotiation.cc:407] Connection header received
0926 02:07:00.685195 (+ 2449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:00.685202 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:00.685298 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:00.685392 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:00.688332 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.689791 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.692660 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.693486 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.693633 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:00.698485 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:00.698492 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:00.698496 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:00.698555 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:00.702680 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:00.704209 (+ 1529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:00.704225 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:00.704230 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:00.704391 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:00.704934 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:00.704939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:00.704944 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:00.705281 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:00.705451 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:00.706294 (+ 843us) server_negotiation.cc:299] Negotiation successful
0926 02:07:00.706539 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":1527,"threads_started":1}
I20260926 02:07:00.708671 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:00.681542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:39677 (local address 127.18.159.131:59611)
0926 02:07:00.682544 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:00.682592 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:00.684746 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:00.686045 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:00.686058 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:00.686980 (+ 922us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:00.688029 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:00.688081 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.690015 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.690022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:00.692365 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:00.692393 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.693746 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.693754 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:00.694027 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:00.694870 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:00.694895 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:00.698112 (+ 3217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:00.702940 (+ 4828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:00.702949 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:00.702965 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:00.703939 (+ 974us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:00.704544 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:00.704550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:00.704554 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:00.704794 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:00.705431 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:00.705440 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:00.705885 (+ 445us) client_negotiation.cc:769] Sending connection context
0926 02:07:00.706186 (+ 301us) client_negotiation.cc:240] Negotiation successful
0926 02:07:00.706793 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":483,"threads_started":1}
I20260926 02:07:00.734607 21194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:00.718676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:45995 (local address 127.18.159.190:39677)
0926 02:07:00.718866 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:00.718871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:00.719438 (+ 567us) server_negotiation.cc:407] Connection header received
0926 02:07:00.719711 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:00.719716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:00.719829 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:00.719943 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:00.721140 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.721895 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.723401 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.723862 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.724021 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:00.727428 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:00.727435 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:00.727438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:00.727474 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:00.731182 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:00.732406 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:00.732414 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:00.732417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:00.732562 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:00.733002 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:00.733007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:00.733009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:00.733315 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:00.733486 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:00.733978 (+ 492us) server_negotiation.cc:299] Negotiation successful
0926 02:07:00.734231 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:07:00.734738 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:00.718636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:39677 (local address 127.18.159.131:45995)
0926 02:07:00.719306 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:00.719323 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:00.719530 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:00.720202 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:00.720208 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:00.720666 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:00.720967 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:00.720976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.722072 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.722077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:00.723234 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:00.723247 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.724104 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.724111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:00.724375 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:00.725439 (+ 1064us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:00.725472 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:00.726827 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:00.731446 (+ 4619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:00.731458 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:00.731463 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:00.732150 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:00.732698 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:00.732703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:00.732706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:00.732851 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:00.733453 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:00.733458 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:00.733659 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 02:07:00.733870 (+ 211us) client_negotiation.cc:240] Negotiation successful
0926 02:07:00.734266 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":389,"threads_started":1}
I20260926 02:07:00.742683 21170 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:37133
I20260926 02:07:00.742705 21305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:37133 every 8 connection(s)
I20260926 02:07:00.744010 21170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:07:00.746634 21306 heartbeater.cc:347] Connected to a master server at 127.18.159.190:39677
I20260926 02:07:00.746910 21306 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:00.747638 21306 heartbeater.cc:511] Master 127.18.159.190:39677 requested a full tablet report, sending...
I20260926 02:07:00.748909 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 21170
I20260926 02:07:00.749118 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:07:00.749418 20819 ts_manager.cc:194] Registered new tserver with Master: 441d63ac043b43ce9e79d6a7f53c5434 (127.18.159.131:37133)
I20260926 02:07:00.749604 19070 external_mini_cluster.cc:1428] Setting key 11f73e01146d0b27ebe1a52edfbb44a5
I20260926 02:07:00.750614 20819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:59611
I20260926 02:07:00.753352 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:07:00 dist-test-slave-03rn krb5kdc[20774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:07:00 dist-test-slave-03rn krb5kdc[20774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:00.818785 21194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:00.801902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42048 (local address 127.18.159.190:39677)
0926 02:07:00.802120 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:00.802125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:00.802702 (+ 577us) server_negotiation.cc:407] Connection header received
0926 02:07:00.802822 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:00.802826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:00.802910 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:00.803013 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:00.804707 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.805539 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.806841 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.807294 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.807407 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:00.812889 (+ 5482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:00.812896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:00.812899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:00.812943 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:00.815606 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:00.816609 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:00.816614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:00.816616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:00.816757 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:00.817231 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:00.817234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:00.817236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:00.817574 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:00.817786 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:00.818287 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 02:07:00.818523 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 02:07:00.823664 20819 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:07:00.830964 20819 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:07:00.874166 21322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:00.846890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57446 (local address 127.18.159.130:38649)
0926 02:07:00.862616 (+ 15726us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:00.862624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:00.862656 (+ 32us) server_negotiation.cc:407] Connection header received
0926 02:07:00.862743 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:00.862750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:00.862968 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:00.863136 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:00.866278 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.867595 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.870925 (+ 3330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.871992 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.872261 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:00.872482 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:00.873560 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0926 02:07:00.873753 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15561,"thread_start_us":15502,"threads_started":1}
I20260926 02:07:00.874325 21324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:00.852628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:34257 (local address 127.0.0.1:39358)
0926 02:07:00.861437 (+ 8809us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:00.861467 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:00.861643 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:00.863489 (+ 1846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:00.863498 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:00.863554 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:00.864338 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:00.864357 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.866971 (+ 2614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.866984 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:00.870547 (+ 3563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:00.870562 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.873143 (+ 2581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.873149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:00.873343 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:00.873380 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 02:07:00.873461 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 02:07:00.873712 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8605,"mutex_wait_us":188,"spinlock_wait_cycles":768,"thread_start_us":8448,"threads_started":1}
I20260926 02:07:00.874506 21319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:00.845949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:37133 (local address 127.0.0.1:50622)
0926 02:07:00.853151 (+ 7202us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:00.853220 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:00.853479 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:00.865642 (+ 12163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:00.865656 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:00.865692 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:00.866048 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:00.866060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.868645 (+ 2585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.868655 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:00.870602 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:00.870620 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.873712 (+ 3092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.873720 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:00.873918 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:00.873942 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 02:07:00.874003 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 02:07:00.874116 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6852,"mutex_wait_us":134,"thread_start_us":686,"threads_started":1}
I20260926 02:07:00.875124 21323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:00.851622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.18.159.129:34257)
0926 02:07:00.862298 (+ 10676us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:00.862312 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:00.862350 (+ 38us) server_negotiation.cc:407] Connection header received
0926 02:07:00.862472 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:00.862483 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:00.862843 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:00.863116 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:00.865143 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.866803 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.870771 (+ 3968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.872702 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.872898 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:00.873080 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:00.874604 (+ 1524us) server_negotiation.cc:299] Negotiation successful
0926 02:07:00.874805 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10457,"thread_start_us":10257,"threads_started":1}
I20260926 02:07:00.875121 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:00.846587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:38649 (local address 127.0.0.1:57446)
0926 02:07:00.859475 (+ 12888us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:00.859508 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:00.859726 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:00.863489 (+ 3763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:00.863498 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:00.863559 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:00.864665 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:00.864679 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.867744 (+ 3065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.867752 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:00.870697 (+ 2945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:00.870710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.872558 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.872565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:00.872808 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:00.872878 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 02:07:00.874565 (+ 1687us) client_negotiation.cc:240] Negotiation successful
0926 02:07:00.874719 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12659,"mutex_wait_us":594,"spinlock_wait_cycles":4992,"thread_start_us":12514,"threads_started":1}
I20260926 02:07:00.875761 21321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:00.846620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50622 (local address 127.18.159.131:37133)
0926 02:07:00.864655 (+ 18035us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:00.864665 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:00.864708 (+ 43us) server_negotiation.cc:407] Connection header received
0926 02:07:00.865020 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:00.865040 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:00.865371 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:00.866308 (+ 937us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:00.866701 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.868306 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.870833 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:00.873454 (+ 2621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:00.874845 (+ 1391us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:00.875121 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:00.875221 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 02:07:00.875425 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17811,"thread_start_us":17872,"threads_started":1}
I20260926 02:07:00.879297 20956 tablet_service.cc:1467] Processing CreateTablet for tablet 36b645b63a7448059c9099a1b85b0063 (DEFAULT_TABLE table=test-table [id=52be807465774a8c8969e9d37c673ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:00.879295 21236 tablet_service.cc:1467] Processing CreateTablet for tablet 36b645b63a7448059c9099a1b85b0063 (DEFAULT_TABLE table=test-table [id=52be807465774a8c8969e9d37c673ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:00.879304 21100 tablet_service.cc:1467] Processing CreateTablet for tablet 36b645b63a7448059c9099a1b85b0063 (DEFAULT_TABLE table=test-table [id=52be807465774a8c8969e9d37c673ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:00.880918 20956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36b645b63a7448059c9099a1b85b0063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:00.880952 21236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36b645b63a7448059c9099a1b85b0063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:00.881289 21100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36b645b63a7448059c9099a1b85b0063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:00.895130 21325 tablet_bootstrap.cc:492] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Bootstrap starting.
I20260926 02:07:00.898980 21325 tablet_bootstrap.cc:654] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:00.900426 21325 log.cc:824] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:00.903195 21327 tablet_bootstrap.cc:492] T 36b645b63a7448059c9099a1b85b0063 P dd710cc670f54fa491a96609dcb96ce9: Bootstrap starting.
I20260926 02:07:00.903195 21326 tablet_bootstrap.cc:492] T 36b645b63a7448059c9099a1b85b0063 P 441d63ac043b43ce9e79d6a7f53c5434: Bootstrap starting.
I20260926 02:07:00.907282 21326 tablet_bootstrap.cc:654] T 36b645b63a7448059c9099a1b85b0063 P 441d63ac043b43ce9e79d6a7f53c5434: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:00.908680 21327 tablet_bootstrap.cc:654] T 36b645b63a7448059c9099a1b85b0063 P dd710cc670f54fa491a96609dcb96ce9: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:00.910085 21327 log.cc:824] T 36b645b63a7448059c9099a1b85b0063 P dd710cc670f54fa491a96609dcb96ce9: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:00.910459 21326 log.cc:824] T 36b645b63a7448059c9099a1b85b0063 P 441d63ac043b43ce9e79d6a7f53c5434: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:00.913023 21325 tablet_bootstrap.cc:492] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: No bootstrap required, opened a new log
I20260926 02:07:00.913367 21325 ts_tablet_manager.cc:1397] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.014s
I20260926 02:07:00.913604 21327 tablet_bootstrap.cc:492] T 36b645b63a7448059c9099a1b85b0063 P dd710cc670f54fa491a96609dcb96ce9: No bootstrap required, opened a new log
I20260926 02:07:00.913731 21326 tablet_bootstrap.cc:492] T 36b645b63a7448059c9099a1b85b0063 P 441d63ac043b43ce9e79d6a7f53c5434: No bootstrap required, opened a new log
I20260926 02:07:00.913909 21327 ts_tablet_manager.cc:1397] T 36b645b63a7448059c9099a1b85b0063 P dd710cc670f54fa491a96609dcb96ce9: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260926 02:07:00.914031 21326 ts_tablet_manager.cc:1397] T 36b645b63a7448059c9099a1b85b0063 P 441d63ac043b43ce9e79d6a7f53c5434: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260926 02:07:00.922520 21326 raft_consensus.cc:348] T 36b645b63a7448059c9099a1b85b0063 P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } } peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } }
I20260926 02:07:00.923003 21326 raft_consensus.cc:374] T 36b645b63a7448059c9099a1b85b0063 P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:00.923105 21326 raft_consensus.cc:729] T 36b645b63a7448059c9099a1b85b0063 P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441d63ac043b43ce9e79d6a7f53c5434, State: Initialized, Role: FOLLOWER
I20260926 02:07:00.923753 21326 consensus_queue.cc:260] T 36b645b63a7448059c9099a1b85b0063 P 441d63ac043b43ce9e79d6a7f53c5434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } } peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } }
I20260926 02:07:00.925046 21306 heartbeater.cc:503] Master 127.18.159.190:39677 was elected leader, sending a full tablet report...
I20260926 02:07:00.924948 21325 raft_consensus.cc:348] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } } peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } }
I20260926 02:07:00.925467 21325 raft_consensus.cc:374] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:00.925596 21325 raft_consensus.cc:729] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d7d8d8cb3b343ee9ddb2a73a5eb3e43, State: Initialized, Role: FOLLOWER
I20260926 02:07:00.925802 21326 ts_tablet_manager.cc:1428] T 36b645b63a7448059c9099a1b85b0063 P 441d63ac043b43ce9e79d6a7f53c5434: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260926 02:07:00.926149 21327 raft_consensus.cc:348] T 36b645b63a7448059c9099a1b85b0063 P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } } peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } }
I20260926 02:07:00.926234 21325 consensus_queue.cc:260] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } } peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } }
I20260926 02:07:00.926539 21327 raft_consensus.cc:374] T 36b645b63a7448059c9099a1b85b0063 P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:00.926641 21327 raft_consensus.cc:729] T 36b645b63a7448059c9099a1b85b0063 P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd710cc670f54fa491a96609dcb96ce9, State: Initialized, Role: FOLLOWER
I20260926 02:07:00.927273 21327 consensus_queue.cc:260] T 36b645b63a7448059c9099a1b85b0063 P dd710cc670f54fa491a96609dcb96ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } } peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } }
I20260926 02:07:00.928272 21325 ts_tablet_manager.cc:1428] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260926 02:07:00.931707 21327 ts_tablet_manager.cc:1428] T 36b645b63a7448059c9099a1b85b0063 P dd710cc670f54fa491a96609dcb96ce9: Time spent starting tablet: real 0.018s user 0.013s sys 0.003s
W20260926 02:07:00.995904 21307 tablet.cc:2369] T 36b645b63a7448059c9099a1b85b0063 P 441d63ac043b43ce9e79d6a7f53c5434: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:07:01.001356 21027 tablet.cc:2369] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:07:01.026942 21332 raft_consensus.cc:482] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:07:01.027284 21332 raft_consensus.cc:504] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } } peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } }
I20260926 02:07:01.029201 21332 leader_election.cc:288] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 441d63ac043b43ce9e79d6a7f53c5434 (127.18.159.131:37133), dd710cc670f54fa491a96609dcb96ce9 (127.18.159.130:38649)
I20260926 02:07:01.037425 21322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:01.029777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:51761 (local address 127.18.159.130:38649)
0926 02:07:01.030084 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:01.030092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:01.030819 (+ 727us) server_negotiation.cc:407] Connection header received
0926 02:07:01.031020 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:01.031025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:01.031110 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:01.031222 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:01.031988 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.032786 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.035000 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.036422 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.036645 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:01.036725 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:01.037012 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 02:07:01.037146 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260926 02:07:01.037588 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:01.029807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:38649 (local address 127.18.159.129:51761)
0926 02:07:01.030683 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:01.030702 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:01.030879 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:01.031251 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:01.031256 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:01.031280 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:01.031747 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:01.031756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.032932 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.032937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:01.034750 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:01.034767 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.036663 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.036669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:01.036844 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:01.036869 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 02:07:01.036983 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 02:07:01.037143 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":3,"thread_start_us":529,"threads_started":1}
I20260926 02:07:01.038194 21321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:01.029580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:44275 (local address 127.18.159.131:37133)
0926 02:07:01.029860 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:01.029870 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:01.030794 (+ 924us) server_negotiation.cc:407] Connection header received
0926 02:07:01.031018 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:01.031022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:01.031083 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:01.031189 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:01.031950 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.033593 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.036199 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.037193 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.037407 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:01.037487 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:01.037898 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 02:07:01.038018 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 02:07:01.038795 21334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:01.029425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:37133 (local address 127.18.159.129:44275)
0926 02:07:01.030637 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:01.030656 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:01.030857 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:01.031215 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:01.031219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:01.031243 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:01.031717 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:01.031731 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.033780 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.033784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:01.035859 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:01.035878 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.037441 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.037454 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:01.037706 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:01.037744 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 02:07:01.038405 (+ 661us) client_negotiation.cc:240] Negotiation successful
0926 02:07:01.038518 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"mutex_wait_us":21,"thread_start_us":639,"threads_started":1}
I20260926 02:07:01.038708 21121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36b645b63a7448059c9099a1b85b0063" candidate_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd710cc670f54fa491a96609dcb96ce9" is_pre_election: true
I20260926 02:07:01.039346 21121 raft_consensus.cc:2454] T 36b645b63a7448059c9099a1b85b0063 P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 in term 0.
I20260926 02:07:01.039906 21256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36b645b63a7448059c9099a1b85b0063" candidate_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441d63ac043b43ce9e79d6a7f53c5434" is_pre_election: true
I20260926 02:07:01.040138 20907 leader_election.cc:302] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d7d8d8cb3b343ee9ddb2a73a5eb3e43, dd710cc670f54fa491a96609dcb96ce9; no voters:
I20260926 02:07:01.040462 21256 raft_consensus.cc:2454] T 36b645b63a7448059c9099a1b85b0063 P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 in term 0.
I20260926 02:07:01.040666 21332 raft_consensus.cc:2781] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:07:01.040822 21332 raft_consensus.cc:482] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:07:01.040943 21332 raft_consensus.cc:3037] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:01.110896 21332 raft_consensus.cc:504] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } } peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } }
I20260926 02:07:01.111908 21332 leader_election.cc:288] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [CANDIDATE]: Term 1 election: Requested vote from peers 441d63ac043b43ce9e79d6a7f53c5434 (127.18.159.131:37133), dd710cc670f54fa491a96609dcb96ce9 (127.18.159.130:38649)
I20260926 02:07:01.112586 21256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36b645b63a7448059c9099a1b85b0063" candidate_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441d63ac043b43ce9e79d6a7f53c5434"
I20260926 02:07:01.112602 21121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36b645b63a7448059c9099a1b85b0063" candidate_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd710cc670f54fa491a96609dcb96ce9"
I20260926 02:07:01.112841 21256 raft_consensus.cc:3037] T 36b645b63a7448059c9099a1b85b0063 P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:01.112874 21121 raft_consensus.cc:3037] T 36b645b63a7448059c9099a1b85b0063 P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:01.115293 21256 raft_consensus.cc:2454] T 36b645b63a7448059c9099a1b85b0063 P 441d63ac043b43ce9e79d6a7f53c5434 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 in term 1.
I20260926 02:07:01.115934 20909 leader_election.cc:302] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441d63ac043b43ce9e79d6a7f53c5434, 4d7d8d8cb3b343ee9ddb2a73a5eb3e43; no voters:
I20260926 02:07:01.116391 21332 raft_consensus.cc:2781] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:01.116876 21332 raft_consensus.cc:686] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 LEADER]: Becoming Leader. State: Replica: 4d7d8d8cb3b343ee9ddb2a73a5eb3e43, State: Running, Role: LEADER
I20260926 02:07:01.117447 21332 consensus_queue.cc:237] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } } peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } }
I20260926 02:07:01.122859 20818 catalog_manager.cc:5495] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 (127.18.159.129). New cstate: current_term: 1 leader_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } health_report { overall_health: UNKNOWN } } }
W20260926 02:07:01.133702 21167 tablet.cc:2369] T 36b645b63a7448059c9099a1b85b0063 P dd710cc670f54fa491a96609dcb96ce9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 02:07:01 dist-test-slave-03rn krb5kdc[20774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388417, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:01.210552 21121 raft_consensus.cc:2454] T 36b645b63a7448059c9099a1b85b0063 P dd710cc670f54fa491a96609dcb96ce9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 in term 1.
I20260926 02:07:01.213743 21194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:01.198673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42060 (local address 127.18.159.190:39677)
0926 02:07:01.198900 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:01.198907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:01.199593 (+ 686us) server_negotiation.cc:407] Connection header received
0926 02:07:01.199731 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:01.199735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:01.199794 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:01.199881 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:01.201224 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.202155 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.203507 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.203966 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.204071 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:01.207710 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:01.207722 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:01.207725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:01.207758 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:01.210797 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:01.211704 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:01.211709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:01.211711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:01.211839 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:01.212257 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:01.212262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:01.212265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:01.212532 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:01.212694 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:01.213104 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 02:07:01.213425 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":91}
I20260926 02:07:01.213778 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:01.198812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:39677 (local address 127.0.0.1:42060)
0926 02:07:01.199472 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:01.199492 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:01.199622 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:01.199923 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:01.199926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:01.200735 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:01.201037 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:01.201052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.202320 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.202324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:01.203355 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:01.203368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.204204 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.204209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:01.204330 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:01.205080 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:01.205099 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:01.207488 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:01.210926 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:01.210944 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:01.210954 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:01.211568 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:01.211959 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:01.211964 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:01.211968 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:01.212106 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:01.212661 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:01.212670 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:01.212852 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 02:07:01.213010 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 02:07:01.213306 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":402,"threads_started":1}
I20260926 02:07:01.219707 20818 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:42060:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 02:07:01.220840 20818 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:07:01.228097 21100 tablet_service.cc:1467] Processing CreateTablet for tablet d40916c780e1489ea966d26375247bf9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08ee23d42abc42e5b7d849d470082ed0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:07:01.227957 20956 tablet_service.cc:1467] Processing CreateTablet for tablet d40916c780e1489ea966d26375247bf9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08ee23d42abc42e5b7d849d470082ed0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:07:01.228446 21236 tablet_service.cc:1467] Processing CreateTablet for tablet d40916c780e1489ea966d26375247bf9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08ee23d42abc42e5b7d849d470082ed0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:07:01.229127 20956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d40916c780e1489ea966d26375247bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:01.229127 21100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d40916c780e1489ea966d26375247bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:01.229355 21236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d40916c780e1489ea966d26375247bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:01.235812 21326 tablet_bootstrap.cc:492] T d40916c780e1489ea966d26375247bf9 P 441d63ac043b43ce9e79d6a7f53c5434: Bootstrap starting.
I20260926 02:07:01.236744 21327 tablet_bootstrap.cc:492] T d40916c780e1489ea966d26375247bf9 P dd710cc670f54fa491a96609dcb96ce9: Bootstrap starting.
I20260926 02:07:01.237005 21325 tablet_bootstrap.cc:492] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Bootstrap starting.
I20260926 02:07:01.237882 21326 tablet_bootstrap.cc:654] T d40916c780e1489ea966d26375247bf9 P 441d63ac043b43ce9e79d6a7f53c5434: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:01.238595 21327 tablet_bootstrap.cc:654] T d40916c780e1489ea966d26375247bf9 P dd710cc670f54fa491a96609dcb96ce9: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:01.238711 21325 tablet_bootstrap.cc:654] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:01.245087 21325 tablet_bootstrap.cc:492] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: No bootstrap required, opened a new log
I20260926 02:07:01.245146 21327 tablet_bootstrap.cc:492] T d40916c780e1489ea966d26375247bf9 P dd710cc670f54fa491a96609dcb96ce9: No bootstrap required, opened a new log
I20260926 02:07:01.245318 21327 ts_tablet_manager.cc:1397] T d40916c780e1489ea966d26375247bf9 P dd710cc670f54fa491a96609dcb96ce9: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260926 02:07:01.245546 21325 ts_tablet_manager.cc:1397] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260926 02:07:01.246313 21327 raft_consensus.cc:348] T d40916c780e1489ea966d26375247bf9 P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:01.246594 21327 raft_consensus.cc:374] T d40916c780e1489ea966d26375247bf9 P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:01.246676 21327 raft_consensus.cc:729] T d40916c780e1489ea966d26375247bf9 P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd710cc670f54fa491a96609dcb96ce9, State: Initialized, Role: FOLLOWER
I20260926 02:07:01.246965 21327 consensus_queue.cc:260] T d40916c780e1489ea966d26375247bf9 P dd710cc670f54fa491a96609dcb96ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:01.247740 21327 ts_tablet_manager.cc:1428] T d40916c780e1489ea966d26375247bf9 P dd710cc670f54fa491a96609dcb96ce9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:07:01.247866 21333 tablet_replica.cc:439] T d40916c780e1489ea966d26375247bf9 P dd710cc670f54fa491a96609dcb96ce9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } } }
I20260926 02:07:01.248473 21333 tablet_replica.cc:442] T d40916c780e1489ea966d26375247bf9 P dd710cc670f54fa491a96609dcb96ce9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:07:01.249852 21325 raft_consensus.cc:348] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:01.250408 21325 raft_consensus.cc:374] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:01.250624 21325 raft_consensus.cc:729] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d7d8d8cb3b343ee9ddb2a73a5eb3e43, State: Initialized, Role: FOLLOWER
I20260926 02:07:01.251072 21325 consensus_queue.cc:260] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:01.251760 21332 tablet_replica.cc:439] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } } }
I20260926 02:07:01.252125 21332 tablet_replica.cc:442] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:07:01.252854 21325 ts_tablet_manager.cc:1428] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20260926 02:07:01.253175 21326 tablet_bootstrap.cc:492] T d40916c780e1489ea966d26375247bf9 P 441d63ac043b43ce9e79d6a7f53c5434: No bootstrap required, opened a new log
I20260926 02:07:01.253398 21326 ts_tablet_manager.cc:1397] T d40916c780e1489ea966d26375247bf9 P 441d63ac043b43ce9e79d6a7f53c5434: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20260926 02:07:01.254343 21326 raft_consensus.cc:348] T d40916c780e1489ea966d26375247bf9 P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:01.254601 21326 raft_consensus.cc:374] T d40916c780e1489ea966d26375247bf9 P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:01.254704 21326 raft_consensus.cc:729] T d40916c780e1489ea966d26375247bf9 P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441d63ac043b43ce9e79d6a7f53c5434, State: Initialized, Role: FOLLOWER
I20260926 02:07:01.255100 21326 consensus_queue.cc:260] T d40916c780e1489ea966d26375247bf9 P 441d63ac043b43ce9e79d6a7f53c5434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:01.255622 21331 tablet_replica.cc:439] T d40916c780e1489ea966d26375247bf9 P 441d63ac043b43ce9e79d6a7f53c5434: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } } }
I20260926 02:07:01.255903 21331 tablet_replica.cc:442] T d40916c780e1489ea966d26375247bf9 P 441d63ac043b43ce9e79d6a7f53c5434: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:07:01.256587 21326 ts_tablet_manager.cc:1428] T d40916c780e1489ea966d26375247bf9 P 441d63ac043b43ce9e79d6a7f53c5434: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 02:07:01.333757 21332 raft_consensus.cc:482] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:07:01.333971 21332 raft_consensus.cc:504] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:01.334714 21332 leader_election.cc:288] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd710cc670f54fa491a96609dcb96ce9 (127.18.159.130:38649), 441d63ac043b43ce9e79d6a7f53c5434 (127.18.159.131:37133)
I20260926 02:07:01.335350 21121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d40916c780e1489ea966d26375247bf9" candidate_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd710cc670f54fa491a96609dcb96ce9" is_pre_election: true
I20260926 02:07:01.335399 21256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d40916c780e1489ea966d26375247bf9" candidate_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441d63ac043b43ce9e79d6a7f53c5434" is_pre_election: true
I20260926 02:07:01.335618 21121 raft_consensus.cc:2454] T d40916c780e1489ea966d26375247bf9 P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 in term 0.
I20260926 02:07:01.335714 21256 raft_consensus.cc:2454] T d40916c780e1489ea966d26375247bf9 P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 in term 0.
I20260926 02:07:01.336118 20907 leader_election.cc:302] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d7d8d8cb3b343ee9ddb2a73a5eb3e43, dd710cc670f54fa491a96609dcb96ce9; no voters:
I20260926 02:07:01.336513 21332 raft_consensus.cc:2781] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:07:01.336640 21332 raft_consensus.cc:482] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:07:01.336802 21332 raft_consensus.cc:3037] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:01.415771 21332 raft_consensus.cc:504] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:01.416673 21332 leader_election.cc:288] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [CANDIDATE]: Term 1 election: Requested vote from peers dd710cc670f54fa491a96609dcb96ce9 (127.18.159.130:38649), 441d63ac043b43ce9e79d6a7f53c5434 (127.18.159.131:37133)
I20260926 02:07:01.417234 21121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d40916c780e1489ea966d26375247bf9" candidate_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd710cc670f54fa491a96609dcb96ce9"
I20260926 02:07:01.417272 21256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d40916c780e1489ea966d26375247bf9" candidate_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441d63ac043b43ce9e79d6a7f53c5434"
I20260926 02:07:01.417531 21121 raft_consensus.cc:3037] T d40916c780e1489ea966d26375247bf9 P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:01.417554 21256 raft_consensus.cc:3037] T d40916c780e1489ea966d26375247bf9 P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:01.517648 21121 raft_consensus.cc:2454] T d40916c780e1489ea966d26375247bf9 P dd710cc670f54fa491a96609dcb96ce9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 in term 1.
I20260926 02:07:01.517743 21256 raft_consensus.cc:2454] T d40916c780e1489ea966d26375247bf9 P 441d63ac043b43ce9e79d6a7f53c5434 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 in term 1.
I20260926 02:07:01.518405 20907 leader_election.cc:302] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 441d63ac043b43ce9e79d6a7f53c5434, 4d7d8d8cb3b343ee9ddb2a73a5eb3e43, dd710cc670f54fa491a96609dcb96ce9; no voters:
I20260926 02:07:01.518800 21332 raft_consensus.cc:2781] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:01.519039 21332 raft_consensus.cc:686] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 LEADER]: Becoming Leader. State: Replica: 4d7d8d8cb3b343ee9ddb2a73a5eb3e43, State: Running, Role: LEADER
I20260926 02:07:01.519313 21332 consensus_queue.cc:237] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:01.521121 21336 tablet_replica.cc:439] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: TxnStatusTablet state changed. Reason: New leader 4d7d8d8cb3b343ee9ddb2a73a5eb3e43. Latest consensus state: current_term: 1 leader_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } } }
I20260926 02:07:01.521456 21336 tablet_replica.cc:442] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:07:01.522137 21348 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 02:07:01.522440 21348 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 02:07:01.522758 20819 catalog_manager.cc:5495] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 (127.18.159.129). New cstate: current_term: 1 leader_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } health_report { overall_health: UNKNOWN } } }
I20260926 02:07:01.551374 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:01.543921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:34257 (local address 127.0.0.1:39374)
0926 02:07:01.544245 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:01.544272 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:01.544466 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:01.545286 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:01.545289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:01.545300 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:07:01.545490 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:01.545498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.547017 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.547025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:01.548116 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:01.548137 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.549277 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.549283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:01.549518 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:01.549598 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:07:01.550772 (+ 1174us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:07:01.550797 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 02:07:01.550886 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 02:07:01.551055 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 02:07:01.551368 21351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:01.544042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39374 (local address 127.18.159.129:34257)
0926 02:07:01.544666 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:01.544672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:01.544694 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:07:01.544802 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:01.544809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:01.544912 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:01.545099 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:07:01.545631 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.546837 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.548345 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.549056 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.549229 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:01.549824 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:07:01.550639 (+ 815us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:07:01.550803 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:01.550949 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 02:07:01.551041 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":290,"threads_started":1}
I20260926 02:07:01.561479 21121 raft_consensus.cc:1261] T d40916c780e1489ea966d26375247bf9 P dd710cc670f54fa491a96609dcb96ce9 [term 1 FOLLOWER]: Refusing update from remote peer 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:01.561483 21256 raft_consensus.cc:1261] T d40916c780e1489ea966d26375247bf9 P 441d63ac043b43ce9e79d6a7f53c5434 [term 1 FOLLOWER]: Refusing update from remote peer 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:01.562659 21332 consensus_queue.cc:1035] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:01.563001 21336 consensus_queue.cc:1035] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:01.576382 21331 tablet_replica.cc:439] T d40916c780e1489ea966d26375247bf9 P 441d63ac043b43ce9e79d6a7f53c5434: TxnStatusTablet state changed. Reason: New leader 4d7d8d8cb3b343ee9ddb2a73a5eb3e43. Latest consensus state: current_term: 1 leader_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } } }
I20260926 02:07:01.576650 21331 tablet_replica.cc:442] T d40916c780e1489ea966d26375247bf9 P 441d63ac043b43ce9e79d6a7f53c5434: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:07:01.578190 21352 mvcc.cc:205] Tried to move back new op lower bound from 7333430974703611904 to 7333430974545727488. Current Snapshot: MvccSnapshot[applied={T|T < 7333430974703611904}]
I20260926 02:07:01.582711 21336 tablet_replica.cc:439] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } } }
I20260926 02:07:01.582998 21336 tablet_replica.cc:442] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:07:01.585577 21353 mvcc.cc:205] Tried to move back new op lower bound from 7333430974703611904 to 7333430974545727488. Current Snapshot: MvccSnapshot[applied={T|T < 7333430974703611904}]
I20260926 02:07:01.594398 21333 tablet_replica.cc:439] T d40916c780e1489ea966d26375247bf9 P dd710cc670f54fa491a96609dcb96ce9: TxnStatusTablet state changed. Reason: New leader 4d7d8d8cb3b343ee9ddb2a73a5eb3e43. Latest consensus state: current_term: 1 leader_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } } }
I20260926 02:07:01.594729 21333 tablet_replica.cc:442] T d40916c780e1489ea966d26375247bf9 P dd710cc670f54fa491a96609dcb96ce9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:07:01.594614 21331 tablet_replica.cc:439] T d40916c780e1489ea966d26375247bf9 P 441d63ac043b43ce9e79d6a7f53c5434: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } } }
I20260926 02:07:01.594842 21331 tablet_replica.cc:442] T d40916c780e1489ea966d26375247bf9 P 441d63ac043b43ce9e79d6a7f53c5434: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:07:01.596369 21332 tablet_replica.cc:439] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } } }
I20260926 02:07:01.596732 21332 tablet_replica.cc:442] T d40916c780e1489ea966d26375247bf9 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:07:01.599718 21333 tablet_replica.cc:439] T d40916c780e1489ea966d26375247bf9 P dd710cc670f54fa491a96609dcb96ce9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } } }
I20260926 02:07:01.599969 21333 tablet_replica.cc:442] T d40916c780e1489ea966d26375247bf9 P dd710cc670f54fa491a96609dcb96ce9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:07:01.608763 21351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:01.600346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.18.159.129:34257)
0926 02:07:01.600640 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:01.600646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:01.602972 (+ 2326us) server_negotiation.cc:407] Connection header received
0926 02:07:01.603152 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:01.603158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:01.603239 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:01.603338 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:07:01.603732 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.604942 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.606638 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.607339 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.607471 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:01.607892 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:07:01.608033 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:07:01.608141 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:01.608365 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 02:07:01.608456 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260926 02:07:01.627173 21336 consensus_queue.cc:1035] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:01.630859 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:01.623289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:34257 (local address 127.18.159.129:50389)
0926 02:07:01.623922 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:01.623937 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:01.624117 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:01.624540 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:01.624545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:01.624554 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:07:01.624918 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:01.624927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.626136 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.626142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:01.628075 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:01.628102 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.629178 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.629187 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:01.629364 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:01.629446 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:07:01.630016 (+ 570us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:07:01.630028 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:07:01.630087 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 02:07:01.630317 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":358,"threads_started":1}
I20260926 02:07:01.632275 21351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:01.623448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:50389 (local address 127.18.159.129:34257)
0926 02:07:01.623645 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:01.623654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:01.624026 (+ 372us) server_negotiation.cc:407] Connection header received
0926 02:07:01.624341 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:01.624345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:01.624419 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:01.624519 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:07:01.625068 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.625963 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.628266 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:01.628966 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:01.629666 (+ 700us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:01.629768 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:07:01.629907 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:07:01.631117 (+ 1210us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:01.631167 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 02:07:01.631252 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 02:07:01.633288 21332 consensus_queue.cc:1035] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:01.714288 20818 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:01.714581 20818 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:01.719004 20818 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24: Sending DeleteTablet for 3 replicas of tablet 36b645b63a7448059c9099a1b85b0063
I20260926 02:07:01.720434 21236 tablet_service.cc:1514] Processing DeleteTablet for tablet 36b645b63a7448059c9099a1b85b0063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:01 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:50622
I20260926 02:07:01.720858 21100 tablet_service.cc:1514] Processing DeleteTablet for tablet 36b645b63a7448059c9099a1b85b0063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:01 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:57446
I20260926 02:07:01.723627 20818 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 02:07:01.724370 21370 tablet_replica.cc:330] T 36b645b63a7448059c9099a1b85b0063 P 441d63ac043b43ce9e79d6a7f53c5434: stopping tablet replica
I20260926 02:07:01.725039 21370 raft_consensus.cc:2229] T 36b645b63a7448059c9099a1b85b0063 P 441d63ac043b43ce9e79d6a7f53c5434 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 02:07:01.725198 20818 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:07:01.725495 21370 raft_consensus.cc:2258] T 36b645b63a7448059c9099a1b85b0063 P 441d63ac043b43ce9e79d6a7f53c5434 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:01.727758 21370 ts_tablet_manager.cc:1905] T 36b645b63a7448059c9099a1b85b0063 P 441d63ac043b43ce9e79d6a7f53c5434: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:01.728091 20956 tablet_service.cc:1514] Processing DeleteTablet for tablet 36b645b63a7448059c9099a1b85b0063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:01 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:39358
I20260926 02:07:01.732525 21372 tablet_replica.cc:330] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: stopping tablet replica
I20260926 02:07:01.733112 21372 raft_consensus.cc:2229] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:07:01.733870 21372 raft_consensus.cc:2258] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:01.735590 21099 tablet_service.cc:1467] Processing CreateTablet for tablet 9602ac52a49546d1b6aa5154edcd032d (DEFAULT_TABLE table=test-table [id=b33feffa496f4f9cb74145765d84bd21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:01.736033 21372 ts_tablet_manager.cc:1905] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:01.736196 21099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9602ac52a49546d1b6aa5154edcd032d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:01.736430 21236 tablet_service.cc:1467] Processing CreateTablet for tablet 9602ac52a49546d1b6aa5154edcd032d (DEFAULT_TABLE table=test-table [id=b33feffa496f4f9cb74145765d84bd21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:01.737203 20956 tablet_service.cc:1467] Processing CreateTablet for tablet 9602ac52a49546d1b6aa5154edcd032d (DEFAULT_TABLE table=test-table [id=b33feffa496f4f9cb74145765d84bd21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:01.737732 20956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9602ac52a49546d1b6aa5154edcd032d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:01.743731 21327 tablet_bootstrap.cc:492] T 9602ac52a49546d1b6aa5154edcd032d P dd710cc670f54fa491a96609dcb96ce9: Bootstrap starting.
I20260926 02:07:01.745754 21327 tablet_bootstrap.cc:654] T 9602ac52a49546d1b6aa5154edcd032d P dd710cc670f54fa491a96609dcb96ce9: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:01.749469 21327 tablet_bootstrap.cc:492] T 9602ac52a49546d1b6aa5154edcd032d P dd710cc670f54fa491a96609dcb96ce9: No bootstrap required, opened a new log
I20260926 02:07:01.749760 21327 ts_tablet_manager.cc:1397] T 9602ac52a49546d1b6aa5154edcd032d P dd710cc670f54fa491a96609dcb96ce9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 02:07:01.749953 21371 tablet_replica.cc:330] T 36b645b63a7448059c9099a1b85b0063 P dd710cc670f54fa491a96609dcb96ce9: stopping tablet replica
I20260926 02:07:01.750435 21371 raft_consensus.cc:2229] T 36b645b63a7448059c9099a1b85b0063 P dd710cc670f54fa491a96609dcb96ce9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:01.750834 21371 raft_consensus.cc:2258] T 36b645b63a7448059c9099a1b85b0063 P dd710cc670f54fa491a96609dcb96ce9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:01.752312 21327 raft_consensus.cc:348] T 9602ac52a49546d1b6aa5154edcd032d P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:01.752682 21327 raft_consensus.cc:374] T 9602ac52a49546d1b6aa5154edcd032d P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:01.752800 21327 raft_consensus.cc:729] T 9602ac52a49546d1b6aa5154edcd032d P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd710cc670f54fa491a96609dcb96ce9, State: Initialized, Role: FOLLOWER
I20260926 02:07:01.753013 21371 ts_tablet_manager.cc:1905] T 36b645b63a7448059c9099a1b85b0063 P dd710cc670f54fa491a96609dcb96ce9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:01.753645 21327 consensus_queue.cc:260] T 9602ac52a49546d1b6aa5154edcd032d P dd710cc670f54fa491a96609dcb96ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:01.754384 21327 ts_tablet_manager.cc:1428] T 9602ac52a49546d1b6aa5154edcd032d P dd710cc670f54fa491a96609dcb96ce9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 02:07:01.825215 21236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9602ac52a49546d1b6aa5154edcd032d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:01.833184 21374 tablet_bootstrap.cc:492] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Bootstrap starting.
I20260926 02:07:01.835592 21374 tablet_bootstrap.cc:654] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:01.836616 21375 tablet_bootstrap.cc:492] T 9602ac52a49546d1b6aa5154edcd032d P 441d63ac043b43ce9e79d6a7f53c5434: Bootstrap starting.
I20260926 02:07:01.838410 21375 tablet_bootstrap.cc:654] T 9602ac52a49546d1b6aa5154edcd032d P 441d63ac043b43ce9e79d6a7f53c5434: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:01.838564 21374 tablet_bootstrap.cc:492] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: No bootstrap required, opened a new log
I20260926 02:07:01.838740 21374 ts_tablet_manager.cc:1397] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260926 02:07:01.839764 21374 raft_consensus.cc:348] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:01.840027 21374 raft_consensus.cc:374] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:01.840132 21374 raft_consensus.cc:729] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d7d8d8cb3b343ee9ddb2a73a5eb3e43, State: Initialized, Role: FOLLOWER
I20260926 02:07:01.840447 21374 consensus_queue.cc:260] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:01.841002 21374 ts_tablet_manager.cc:1428] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:07:01.841629 21375 tablet_bootstrap.cc:492] T 9602ac52a49546d1b6aa5154edcd032d P 441d63ac043b43ce9e79d6a7f53c5434: No bootstrap required, opened a new log
I20260926 02:07:01.841794 21375 ts_tablet_manager.cc:1397] T 9602ac52a49546d1b6aa5154edcd032d P 441d63ac043b43ce9e79d6a7f53c5434: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260926 02:07:01.842602 21375 raft_consensus.cc:348] T 9602ac52a49546d1b6aa5154edcd032d P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:01.842852 21375 raft_consensus.cc:374] T 9602ac52a49546d1b6aa5154edcd032d P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:01.842949 21375 raft_consensus.cc:729] T 9602ac52a49546d1b6aa5154edcd032d P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441d63ac043b43ce9e79d6a7f53c5434, State: Initialized, Role: FOLLOWER
I20260926 02:07:01.843205 21375 consensus_queue.cc:260] T 9602ac52a49546d1b6aa5154edcd032d P 441d63ac043b43ce9e79d6a7f53c5434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:01.843823 21375 ts_tablet_manager.cc:1428] T 9602ac52a49546d1b6aa5154edcd032d P 441d63ac043b43ce9e79d6a7f53c5434: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 02:07:01.884919 21367 raft_consensus.cc:482] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:07:01.885183 21367 raft_consensus.cc:504] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:01.886126 21367 leader_election.cc:288] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd710cc670f54fa491a96609dcb96ce9 (127.18.159.130:38649), 441d63ac043b43ce9e79d6a7f53c5434 (127.18.159.131:37133)
I20260926 02:07:01.886638 21121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9602ac52a49546d1b6aa5154edcd032d" candidate_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd710cc670f54fa491a96609dcb96ce9" is_pre_election: true
I20260926 02:07:01.886902 21121 raft_consensus.cc:2454] T 9602ac52a49546d1b6aa5154edcd032d P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 in term 0.
I20260926 02:07:01.887156 21255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9602ac52a49546d1b6aa5154edcd032d" candidate_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441d63ac043b43ce9e79d6a7f53c5434" is_pre_election: true
I20260926 02:07:01.887393 20907 leader_election.cc:302] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d7d8d8cb3b343ee9ddb2a73a5eb3e43, dd710cc670f54fa491a96609dcb96ce9; no voters:
I20260926 02:07:01.887408 21255 raft_consensus.cc:2454] T 9602ac52a49546d1b6aa5154edcd032d P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 in term 0.
I20260926 02:07:01.887671 21367 raft_consensus.cc:2781] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:07:01.887782 21367 raft_consensus.cc:482] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:07:01.887871 21367 raft_consensus.cc:3037] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:01.924347 21370 ts_tablet_manager.cc:1918] T 36b645b63a7448059c9099a1b85b0063 P 441d63ac043b43ce9e79d6a7f53c5434: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:07:01.924346 21372 ts_tablet_manager.cc:1918] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:07:01.924561 21370 log.cc:1196] T 36b645b63a7448059c9099a1b85b0063 P 441d63ac043b43ce9e79d6a7f53c5434: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/36b645b63a7448059c9099a1b85b0063
I20260926 02:07:01.924562 21372 log.cc:1196] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/36b645b63a7448059c9099a1b85b0063
I20260926 02:07:01.925135 21372 ts_tablet_manager.cc:1939] T 36b645b63a7448059c9099a1b85b0063 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Deleting consensus metadata
I20260926 02:07:01.925150 21370 ts_tablet_manager.cc:1939] T 36b645b63a7448059c9099a1b85b0063 P 441d63ac043b43ce9e79d6a7f53c5434: Deleting consensus metadata
I20260926 02:07:01.925598 21371 ts_tablet_manager.cc:1918] T 36b645b63a7448059c9099a1b85b0063 P dd710cc670f54fa491a96609dcb96ce9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:07:01.925784 21371 log.cc:1196] T 36b645b63a7448059c9099a1b85b0063 P dd710cc670f54fa491a96609dcb96ce9: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/36b645b63a7448059c9099a1b85b0063
I20260926 02:07:01.926298 21371 ts_tablet_manager.cc:1939] T 36b645b63a7448059c9099a1b85b0063 P dd710cc670f54fa491a96609dcb96ce9: Deleting consensus metadata
I20260926 02:07:01.926555 21367 raft_consensus.cc:504] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:01.927409 21367 leader_election.cc:288] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [CANDIDATE]: Term 1 election: Requested vote from peers dd710cc670f54fa491a96609dcb96ce9 (127.18.159.130:38649), 441d63ac043b43ce9e79d6a7f53c5434 (127.18.159.131:37133)
I20260926 02:07:01.928485 21255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9602ac52a49546d1b6aa5154edcd032d" candidate_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441d63ac043b43ce9e79d6a7f53c5434"
I20260926 02:07:01.928710 21255 raft_consensus.cc:3037] T 9602ac52a49546d1b6aa5154edcd032d P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:01.928838 21121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9602ac52a49546d1b6aa5154edcd032d" candidate_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd710cc670f54fa491a96609dcb96ce9"
I20260926 02:07:01.929019 21121 raft_consensus.cc:3037] T 9602ac52a49546d1b6aa5154edcd032d P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:01.930042 20806 catalog_manager.cc:4841] TS 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 (127.18.159.129:34257): tablet 36b645b63a7448059c9099a1b85b0063 (table test-table [id=52be807465774a8c8969e9d37c673ca5]) successfully deleted
I20260926 02:07:01.932243 20806 catalog_manager.cc:4841] TS 441d63ac043b43ce9e79d6a7f53c5434 (127.18.159.131:37133): tablet 36b645b63a7448059c9099a1b85b0063 (table test-table [id=52be807465774a8c8969e9d37c673ca5]) successfully deleted
I20260926 02:07:01.940295 20804 catalog_manager.cc:4841] TS dd710cc670f54fa491a96609dcb96ce9 (127.18.159.130:38649): tablet 36b645b63a7448059c9099a1b85b0063 (table test-table [id=52be807465774a8c8969e9d37c673ca5]) successfully deleted
I20260926 02:07:02.026973 21255 raft_consensus.cc:2454] T 9602ac52a49546d1b6aa5154edcd032d P 441d63ac043b43ce9e79d6a7f53c5434 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 in term 1.
I20260926 02:07:02.026962 21121 raft_consensus.cc:2454] T 9602ac52a49546d1b6aa5154edcd032d P dd710cc670f54fa491a96609dcb96ce9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 in term 1.
I20260926 02:07:02.027674 20909 leader_election.cc:302] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441d63ac043b43ce9e79d6a7f53c5434, 4d7d8d8cb3b343ee9ddb2a73a5eb3e43; no voters:
I20260926 02:07:02.028218 21367 raft_consensus.cc:2781] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:02.028489 21367 raft_consensus.cc:686] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 LEADER]: Becoming Leader. State: Replica: 4d7d8d8cb3b343ee9ddb2a73a5eb3e43, State: Running, Role: LEADER
I20260926 02:07:02.028851 21367 consensus_queue.cc:237] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:02.034737 20818 catalog_manager.cc:5495] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 (127.18.159.129). New cstate: current_term: 1 leader_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } health_report { overall_health: UNKNOWN } } }
I20260926 02:07:02.073915 21121 raft_consensus.cc:1261] T 9602ac52a49546d1b6aa5154edcd032d P dd710cc670f54fa491a96609dcb96ce9 [term 1 FOLLOWER]: Refusing update from remote peer 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:02.073982 21255 raft_consensus.cc:1261] T 9602ac52a49546d1b6aa5154edcd032d P 441d63ac043b43ce9e79d6a7f53c5434 [term 1 FOLLOWER]: Refusing update from remote peer 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:02.074522 21367 consensus_queue.cc:1035] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:02.074868 21336 consensus_queue.cc:1035] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 02:07:02.104373 21352 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 02:07:03.697924 20818 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:03.698249 20818 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:03.702883 20818 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24: Sending DeleteTablet for 3 replicas of tablet 9602ac52a49546d1b6aa5154edcd032d
I20260926 02:07:03.703847 20955 tablet_service.cc:1514] Processing DeleteTablet for tablet 9602ac52a49546d1b6aa5154edcd032d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:03 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:39358
I20260926 02:07:03.703836 21100 tablet_service.cc:1514] Processing DeleteTablet for tablet 9602ac52a49546d1b6aa5154edcd032d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:03 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:57446
I20260926 02:07:03.704689 21400 tablet_replica.cc:330] T 9602ac52a49546d1b6aa5154edcd032d P dd710cc670f54fa491a96609dcb96ce9: stopping tablet replica
I20260926 02:07:03.705024 21400 raft_consensus.cc:2229] T 9602ac52a49546d1b6aa5154edcd032d P dd710cc670f54fa491a96609dcb96ce9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:03.705332 21400 raft_consensus.cc:2258] T 9602ac52a49546d1b6aa5154edcd032d P dd710cc670f54fa491a96609dcb96ce9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:03.706632 21400 ts_tablet_manager.cc:1905] T 9602ac52a49546d1b6aa5154edcd032d P dd710cc670f54fa491a96609dcb96ce9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:03.708353 21236 tablet_service.cc:1514] Processing DeleteTablet for tablet 9602ac52a49546d1b6aa5154edcd032d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:03 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:50622
I20260926 02:07:03.709002 21402 tablet_replica.cc:330] T 9602ac52a49546d1b6aa5154edcd032d P 441d63ac043b43ce9e79d6a7f53c5434: stopping tablet replica
I20260926 02:07:03.709270 21402 raft_consensus.cc:2229] T 9602ac52a49546d1b6aa5154edcd032d P 441d63ac043b43ce9e79d6a7f53c5434 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:03.709553 21402 raft_consensus.cc:2258] T 9602ac52a49546d1b6aa5154edcd032d P 441d63ac043b43ce9e79d6a7f53c5434 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:03.710680 21402 ts_tablet_manager.cc:1905] T 9602ac52a49546d1b6aa5154edcd032d P 441d63ac043b43ce9e79d6a7f53c5434: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:03.717016 21400 ts_tablet_manager.cc:1918] T 9602ac52a49546d1b6aa5154edcd032d P dd710cc670f54fa491a96609dcb96ce9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:07:03.717378 21400 log.cc:1196] T 9602ac52a49546d1b6aa5154edcd032d P dd710cc670f54fa491a96609dcb96ce9: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/9602ac52a49546d1b6aa5154edcd032d
I20260926 02:07:03.718101 21400 ts_tablet_manager.cc:1939] T 9602ac52a49546d1b6aa5154edcd032d P dd710cc670f54fa491a96609dcb96ce9: Deleting consensus metadata
I20260926 02:07:03.718966 21402 ts_tablet_manager.cc:1918] T 9602ac52a49546d1b6aa5154edcd032d P 441d63ac043b43ce9e79d6a7f53c5434: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:07:03.719125 21402 log.cc:1196] T 9602ac52a49546d1b6aa5154edcd032d P 441d63ac043b43ce9e79d6a7f53c5434: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/9602ac52a49546d1b6aa5154edcd032d
I20260926 02:07:03.719251 21401 tablet_replica.cc:330] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: stopping tablet replica
I20260926 02:07:03.719461 21401 raft_consensus.cc:2229] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:07:03.719614 21402 ts_tablet_manager.cc:1939] T 9602ac52a49546d1b6aa5154edcd032d P 441d63ac043b43ce9e79d6a7f53c5434: Deleting consensus metadata
I20260926 02:07:03.719856 21401 raft_consensus.cc:2258] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:03.719926 20804 catalog_manager.cc:4841] TS dd710cc670f54fa491a96609dcb96ce9 (127.18.159.130:38649): tablet 9602ac52a49546d1b6aa5154edcd032d (table test-table [id=b33feffa496f4f9cb74145765d84bd21]) successfully deleted
I20260926 02:07:03.720706 21401 ts_tablet_manager.cc:1905] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:03.721101 20806 catalog_manager.cc:4841] TS 441d63ac043b43ce9e79d6a7f53c5434 (127.18.159.131:37133): tablet 9602ac52a49546d1b6aa5154edcd032d (table test-table [id=b33feffa496f4f9cb74145765d84bd21]) successfully deleted
I20260926 02:07:03.726306 21401 ts_tablet_manager.cc:1918] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:07:03.726468 21401 log.cc:1196] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/9602ac52a49546d1b6aa5154edcd032d
I20260926 02:07:03.726907 21401 ts_tablet_manager.cc:1939] T 9602ac52a49546d1b6aa5154edcd032d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Deleting consensus metadata
I20260926 02:07:03.728057 20806 catalog_manager.cc:4841] TS 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 (127.18.159.129:34257): tablet 9602ac52a49546d1b6aa5154edcd032d (table test-table [id=b33feffa496f4f9cb74145765d84bd21]) successfully deleted
Sep 26 02:07:03 dist-test-slave-03rn krb5kdc[20774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388423, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 02:07:03 dist-test-slave-03rn krb5kdc[20774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388423, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:03.773563 21410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:03.745744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.18.159.190:39677)
0926 02:07:03.752127 (+ 6383us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:03.752132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:03.752204 (+ 72us) server_negotiation.cc:407] Connection header received
0926 02:07:03.752276 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:03.752280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:03.752365 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:03.752494 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:03.754063 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:03.754959 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:03.756401 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:03.756854 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:03.757016 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:03.768357 (+ 11341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:03.768363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:03.768367 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:03.768411 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:03.770953 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:03.771660 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:03.771664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:03.771666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:03.771812 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:03.772241 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:03.772245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:03.772248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:03.772551 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:03.772752 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:03.773104 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 02:07:03.773287 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6268,"thread_start_us":6070,"threads_started":1}
I20260926 02:07:03.777693 20818 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:07:03.778626 20818 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:07:03.785229 20955 tablet_service.cc:1467] Processing CreateTablet for tablet af586d65708a451689cbce278ad0067d (DEFAULT_TABLE table=test-table [id=955a1089bd6e44bab0ca9e7e66139526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:03.785318 21100 tablet_service.cc:1467] Processing CreateTablet for tablet af586d65708a451689cbce278ad0067d (DEFAULT_TABLE table=test-table [id=955a1089bd6e44bab0ca9e7e66139526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:03.785408 21236 tablet_service.cc:1467] Processing CreateTablet for tablet af586d65708a451689cbce278ad0067d (DEFAULT_TABLE table=test-table [id=955a1089bd6e44bab0ca9e7e66139526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:03.785730 21100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af586d65708a451689cbce278ad0067d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:03.785764 20955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af586d65708a451689cbce278ad0067d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:03.785776 21236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af586d65708a451689cbce278ad0067d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:03.793072 21412 tablet_bootstrap.cc:492] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9: Bootstrap starting.
I20260926 02:07:03.793610 21411 tablet_bootstrap.cc:492] T af586d65708a451689cbce278ad0067d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Bootstrap starting.
I20260926 02:07:03.794061 21413 tablet_bootstrap.cc:492] T af586d65708a451689cbce278ad0067d P 441d63ac043b43ce9e79d6a7f53c5434: Bootstrap starting.
I20260926 02:07:03.795710 21411 tablet_bootstrap.cc:654] T af586d65708a451689cbce278ad0067d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:03.795964 21412 tablet_bootstrap.cc:654] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:03.796482 21413 tablet_bootstrap.cc:654] T af586d65708a451689cbce278ad0067d P 441d63ac043b43ce9e79d6a7f53c5434: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:03.798125 21411 tablet_bootstrap.cc:492] T af586d65708a451689cbce278ad0067d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: No bootstrap required, opened a new log
I20260926 02:07:03.798298 21411 ts_tablet_manager.cc:1397] T af586d65708a451689cbce278ad0067d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260926 02:07:03.798626 21413 tablet_bootstrap.cc:492] T af586d65708a451689cbce278ad0067d P 441d63ac043b43ce9e79d6a7f53c5434: No bootstrap required, opened a new log
I20260926 02:07:03.798782 21413 ts_tablet_manager.cc:1397] T af586d65708a451689cbce278ad0067d P 441d63ac043b43ce9e79d6a7f53c5434: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 02:07:03.799069 21411 raft_consensus.cc:348] T af586d65708a451689cbce278ad0067d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:03.799288 21411 raft_consensus.cc:374] T af586d65708a451689cbce278ad0067d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:03.799396 21411 raft_consensus.cc:729] T af586d65708a451689cbce278ad0067d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d7d8d8cb3b343ee9ddb2a73a5eb3e43, State: Initialized, Role: FOLLOWER
I20260926 02:07:03.799497 21413 raft_consensus.cc:348] T af586d65708a451689cbce278ad0067d P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:03.799755 21413 raft_consensus.cc:374] T af586d65708a451689cbce278ad0067d P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:03.799642 21411 consensus_queue.cc:260] T af586d65708a451689cbce278ad0067d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:03.799854 21413 raft_consensus.cc:729] T af586d65708a451689cbce278ad0067d P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441d63ac043b43ce9e79d6a7f53c5434, State: Initialized, Role: FOLLOWER
I20260926 02:07:03.800112 21413 consensus_queue.cc:260] T af586d65708a451689cbce278ad0067d P 441d63ac043b43ce9e79d6a7f53c5434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:03.800627 21412 tablet_bootstrap.cc:492] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9: No bootstrap required, opened a new log
I20260926 02:07:03.800696 21411 ts_tablet_manager.cc:1428] T af586d65708a451689cbce278ad0067d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 02:07:03.800844 21412 ts_tablet_manager.cc:1397] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260926 02:07:03.801824 21413 ts_tablet_manager.cc:1428] T af586d65708a451689cbce278ad0067d P 441d63ac043b43ce9e79d6a7f53c5434: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 02:07:03.801735 21412 raft_consensus.cc:348] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:03.802016 21412 raft_consensus.cc:374] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:03.802114 21412 raft_consensus.cc:729] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd710cc670f54fa491a96609dcb96ce9, State: Initialized, Role: FOLLOWER
I20260926 02:07:03.802350 21412 consensus_queue.cc:260] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:03.803545 21412 ts_tablet_manager.cc:1428] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 02:07:03.977692 21418 raft_consensus.cc:482] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:07:03.977983 21418 raft_consensus.cc:504] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:03.979678 21418 leader_election.cc:288] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 (127.18.159.129:34257), 441d63ac043b43ce9e79d6a7f53c5434 (127.18.159.131:37133)
I20260926 02:07:03.988461 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:03.979904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:34257 (local address 127.18.159.130:36279)
0926 02:07:03.980637 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:03.980662 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:03.980861 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:03.981671 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:03.981679 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:03.981720 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:03.982244 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:03.982261 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:03.984212 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:03.984218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:03.986097 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:03.986112 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:03.987507 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:03.987514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:03.987741 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:03.987790 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 02:07:03.987917 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 02:07:03.988079 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":1,"thread_start_us":359,"threads_started":1}
I20260926 02:07:03.988636 21420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:03.980121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:36279 (local address 127.18.159.129:34257)
0926 02:07:03.981100 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:03.981110 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:03.981145 (+ 35us) server_negotiation.cc:407] Connection header received
0926 02:07:03.981249 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:03.981259 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:03.981365 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:03.982466 (+ 1101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:03.982844 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:03.984051 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:03.986403 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:03.987249 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:03.987413 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:03.987469 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:03.988052 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 02:07:03.988266 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":366,"threads_started":1}
I20260926 02:07:03.989876 20976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af586d65708a451689cbce278ad0067d" candidate_uuid: "dd710cc670f54fa491a96609dcb96ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" is_pre_election: true
I20260926 02:07:03.990489 20976 raft_consensus.cc:2454] T af586d65708a451689cbce278ad0067d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd710cc670f54fa491a96609dcb96ce9 in term 0.
I20260926 02:07:03.991204 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:03.980945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:60749 (local address 127.18.159.131:37133)
0926 02:07:03.983841 (+ 2896us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:03.983846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:03.983870 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:07:03.983932 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:03.983937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:03.984008 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:03.984140 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:03.984915 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:03.985749 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:03.988072 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:03.989829 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:03.989986 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:03.990041 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:03.990602 (+ 561us) server_negotiation.cc:299] Negotiation successful
0926 02:07:03.990807 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2766,"thread_start_us":2605,"threads_started":1}
I20260926 02:07:03.991204 21421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:03.980752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:37133 (local address 127.18.159.130:60749)
0926 02:07:03.983621 (+ 2869us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:03.983643 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:03.983764 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:03.984212 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:03.984217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:03.984242 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:03.984704 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:03.984715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:03.985917 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:03.985924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:03.987854 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:03.987869 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:03.990174 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:03.990183 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:03.990366 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:03.990402 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:07:03.990530 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 02:07:03.990706 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2689,"thread_start_us":2446,"threads_started":1}
I20260926 02:07:03.991258 21049 leader_election.cc:302] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d7d8d8cb3b343ee9ddb2a73a5eb3e43, dd710cc670f54fa491a96609dcb96ce9; no voters:
I20260926 02:07:03.991798 21418 raft_consensus.cc:2781] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:07:03.991952 21418 raft_consensus.cc:482] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:07:03.992053 21418 raft_consensus.cc:3037] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:03.992434 21256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af586d65708a451689cbce278ad0067d" candidate_uuid: "dd710cc670f54fa491a96609dcb96ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441d63ac043b43ce9e79d6a7f53c5434" is_pre_election: true
I20260926 02:07:03.992812 21256 raft_consensus.cc:2454] T af586d65708a451689cbce278ad0067d P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd710cc670f54fa491a96609dcb96ce9 in term 0.
I20260926 02:07:03.995396 21418 raft_consensus.cc:504] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:03.996237 21418 leader_election.cc:288] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 [CANDIDATE]: Term 1 election: Requested vote from peers 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 (127.18.159.129:34257), 441d63ac043b43ce9e79d6a7f53c5434 (127.18.159.131:37133)
I20260926 02:07:03.996939 20976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af586d65708a451689cbce278ad0067d" candidate_uuid: "dd710cc670f54fa491a96609dcb96ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43"
I20260926 02:07:03.996990 21256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af586d65708a451689cbce278ad0067d" candidate_uuid: "dd710cc670f54fa491a96609dcb96ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441d63ac043b43ce9e79d6a7f53c5434"
I20260926 02:07:03.997211 20976 raft_consensus.cc:3037] T af586d65708a451689cbce278ad0067d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:03.997232 21256 raft_consensus.cc:3037] T af586d65708a451689cbce278ad0067d P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:04.000459 21256 raft_consensus.cc:2454] T af586d65708a451689cbce278ad0067d P 441d63ac043b43ce9e79d6a7f53c5434 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd710cc670f54fa491a96609dcb96ce9 in term 1.
I20260926 02:07:04.000473 20976 raft_consensus.cc:2454] T af586d65708a451689cbce278ad0067d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd710cc670f54fa491a96609dcb96ce9 in term 1.
I20260926 02:07:04.001027 21049 leader_election.cc:302] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d7d8d8cb3b343ee9ddb2a73a5eb3e43, dd710cc670f54fa491a96609dcb96ce9; no voters:
I20260926 02:07:04.001379 21418 raft_consensus.cc:2781] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:04.001912 21418 raft_consensus.cc:686] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 [term 1 LEADER]: Becoming Leader. State: Replica: dd710cc670f54fa491a96609dcb96ce9, State: Running, Role: LEADER
I20260926 02:07:04.002321 21418 consensus_queue.cc:237] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:04.007313 20818 catalog_manager.cc:5495] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd710cc670f54fa491a96609dcb96ce9 (127.18.159.130). New cstate: current_term: 1 leader_uuid: "dd710cc670f54fa491a96609dcb96ce9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } health_report { overall_health: UNKNOWN } } }
I20260926 02:07:04.050586 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:04.043494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57454 (local address 127.18.159.130:38649)
0926 02:07:04.044325 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:04.044329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:04.044352 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:07:04.044444 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:04.044448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:04.044528 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:04.044806 (+ 278us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:07:04.045414 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:04.046356 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:04.047839 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:04.048315 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:04.048422 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:04.048971 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:07:04.049654 (+ 683us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:07:04.049786 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:04.050055 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 02:07:04.050224 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":419,"threads_started":1}
I20260926 02:07:04.067771 21429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:04.061464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:34257 (local address 127.18.159.130:58361)
0926 02:07:04.062089 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:04.062104 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:04.062295 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:04.062609 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:04.062613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:04.062624 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:07:04.063442 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:04.063452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:04.064545 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:04.064549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:04.065682 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:04.065693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:04.066456 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:04.066497 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:04.066628 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:04.066691 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:07:04.067216 (+ 525us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:07:04.067227 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 02:07:04.067319 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 02:07:04.067440 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":318,"threads_started":1}
I20260926 02:07:04.067912 21420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:04.061635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:58361 (local address 127.18.159.129:34257)
0926 02:07:04.061891 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:04.061897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:04.062211 (+ 314us) server_negotiation.cc:407] Connection header received
0926 02:07:04.062425 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:04.062435 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:04.062496 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:04.062584 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:07:04.063589 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:04.064392 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:04.065825 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:04.066259 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:04.066370 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:04.066906 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:07:04.067025 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:07:04.067107 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:04.067373 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 02:07:04.067527 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 02:07:04.077435 21256 raft_consensus.cc:1261] T af586d65708a451689cbce278ad0067d P 441d63ac043b43ce9e79d6a7f53c5434 [term 1 FOLLOWER]: Refusing update from remote peer dd710cc670f54fa491a96609dcb96ce9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:04.077698 20976 raft_consensus.cc:1261] T af586d65708a451689cbce278ad0067d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 FOLLOWER]: Refusing update from remote peer dd710cc670f54fa491a96609dcb96ce9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:04.078296 21423 consensus_queue.cc:1035] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:04.078768 21418 consensus_queue.cc:1035] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:04.112917 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:04.106403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:59711 (local address 127.18.159.130:38649)
0926 02:07:04.106585 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:04.106592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:04.107249 (+ 657us) server_negotiation.cc:407] Connection header received
0926 02:07:04.107445 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:04.107450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:04.107522 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:04.107607 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:07:04.108098 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:04.109082 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:04.111047 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:04.111634 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:04.111793 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:04.112251 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:07:04.112368 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:07:04.112471 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:04.112596 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 02:07:04.112683 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:07:04.113121 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:04.106424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:38649 (local address 127.18.159.129:59711)
0926 02:07:04.107047 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:04.107062 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:04.107319 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:04.107639 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:04.107644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:04.107657 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:07:04.107954 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:04.107964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:04.109254 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:04.109262 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:04.110908 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:04.110923 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:04.111837 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:04.111842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:04.111965 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:04.112003 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:07:04.112475 (+ 472us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:07:04.112487 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:07:04.112570 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 02:07:04.112732 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":352,"threads_started":1}
I20260926 02:07:04.150619 20818 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:04.150894 20818 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:04.154320 20818 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24: Sending DeleteTablet for 3 replicas of tablet af586d65708a451689cbce278ad0067d
I20260926 02:07:04.155345 21100 tablet_service.cc:1514] Processing DeleteTablet for tablet af586d65708a451689cbce278ad0067d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:04 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:57446
I20260926 02:07:04.155390 21236 tablet_service.cc:1514] Processing DeleteTablet for tablet af586d65708a451689cbce278ad0067d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:04 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:50622
I20260926 02:07:04.155675 21400 tablet_replica.cc:330] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9: stopping tablet replica
I20260926 02:07:04.155716 21402 tablet_replica.cc:330] T af586d65708a451689cbce278ad0067d P 441d63ac043b43ce9e79d6a7f53c5434: stopping tablet replica
I20260926 02:07:04.155889 21400 raft_consensus.cc:2229] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:07:04.155943 21402 raft_consensus.cc:2229] T af586d65708a451689cbce278ad0067d P 441d63ac043b43ce9e79d6a7f53c5434 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:04.156208 21402 raft_consensus.cc:2258] T af586d65708a451689cbce278ad0067d P 441d63ac043b43ce9e79d6a7f53c5434 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:04.156348 20955 tablet_service.cc:1514] Processing DeleteTablet for tablet af586d65708a451689cbce278ad0067d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:04 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:39358
I20260926 02:07:04.156481 21400 raft_consensus.cc:2258] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:04.156702 21401 tablet_replica.cc:330] T af586d65708a451689cbce278ad0067d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: stopping tablet replica
I20260926 02:07:04.156625 20818 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 02:07:04.156926 21401 raft_consensus.cc:2229] T af586d65708a451689cbce278ad0067d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:04.157120 21401 raft_consensus.cc:2258] T af586d65708a451689cbce278ad0067d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 02:07:04.157364 20818 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:07:04.158363 21401 ts_tablet_manager.cc:1905] T af586d65708a451689cbce278ad0067d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:04.158622 21400 ts_tablet_manager.cc:1905] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:04.163590 21402 ts_tablet_manager.cc:1905] T af586d65708a451689cbce278ad0067d P 441d63ac043b43ce9e79d6a7f53c5434: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:04.166781 21236 tablet_service.cc:1467] Processing CreateTablet for tablet 6ec64868df554c0c87c032a19c91d800 (DEFAULT_TABLE table=test-table [id=60465baeb77a4bd1ba8cf96c7c73b8ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:04.167933 21236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ec64868df554c0c87c032a19c91d800. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:04.168793 21100 tablet_service.cc:1467] Processing CreateTablet for tablet 6ec64868df554c0c87c032a19c91d800 (DEFAULT_TABLE table=test-table [id=60465baeb77a4bd1ba8cf96c7c73b8ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:04.169227 21100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ec64868df554c0c87c032a19c91d800. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:04.171993 21401 ts_tablet_manager.cc:1918] T af586d65708a451689cbce278ad0067d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:07:04.172231 21401 log.cc:1196] T af586d65708a451689cbce278ad0067d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/af586d65708a451689cbce278ad0067d
I20260926 02:07:04.172667 21401 ts_tablet_manager.cc:1939] T af586d65708a451689cbce278ad0067d P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Deleting consensus metadata
I20260926 02:07:04.172945 20955 tablet_service.cc:1467] Processing CreateTablet for tablet 6ec64868df554c0c87c032a19c91d800 (DEFAULT_TABLE table=test-table [id=60465baeb77a4bd1ba8cf96c7c73b8ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:04.173393 20955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ec64868df554c0c87c032a19c91d800. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:04.176005 21402 ts_tablet_manager.cc:1918] T af586d65708a451689cbce278ad0067d P 441d63ac043b43ce9e79d6a7f53c5434: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:07:04.176218 21402 log.cc:1196] T af586d65708a451689cbce278ad0067d P 441d63ac043b43ce9e79d6a7f53c5434: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/af586d65708a451689cbce278ad0067d
I20260926 02:07:04.176653 21402 ts_tablet_manager.cc:1939] T af586d65708a451689cbce278ad0067d P 441d63ac043b43ce9e79d6a7f53c5434: Deleting consensus metadata
I20260926 02:07:04.177785 21412 tablet_bootstrap.cc:492] T 6ec64868df554c0c87c032a19c91d800 P dd710cc670f54fa491a96609dcb96ce9: Bootstrap starting.
I20260926 02:07:04.177770 20806 catalog_manager.cc:4841] TS 441d63ac043b43ce9e79d6a7f53c5434 (127.18.159.131:37133): tablet af586d65708a451689cbce278ad0067d (table test-table [id=955a1089bd6e44bab0ca9e7e66139526]) successfully deleted
I20260926 02:07:04.179466 20806 catalog_manager.cc:4841] TS 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 (127.18.159.129:34257): tablet af586d65708a451689cbce278ad0067d (table test-table [id=955a1089bd6e44bab0ca9e7e66139526]) successfully deleted
I20260926 02:07:04.180326 21412 tablet_bootstrap.cc:654] T 6ec64868df554c0c87c032a19c91d800 P dd710cc670f54fa491a96609dcb96ce9: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:04.181953 21413 tablet_bootstrap.cc:492] T 6ec64868df554c0c87c032a19c91d800 P 441d63ac043b43ce9e79d6a7f53c5434: Bootstrap starting.
I20260926 02:07:04.182524 21411 tablet_bootstrap.cc:492] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Bootstrap starting.
I20260926 02:07:04.183631 21400 ts_tablet_manager.cc:1918] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:07:04.183804 21400 log.cc:1196] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/af586d65708a451689cbce278ad0067d
I20260926 02:07:04.184194 21413 tablet_bootstrap.cc:654] T 6ec64868df554c0c87c032a19c91d800 P 441d63ac043b43ce9e79d6a7f53c5434: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:04.184212 21411 tablet_bootstrap.cc:654] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:04.184338 21400 ts_tablet_manager.cc:1939] T af586d65708a451689cbce278ad0067d P dd710cc670f54fa491a96609dcb96ce9: Deleting consensus metadata
I20260926 02:07:04.185541 20804 catalog_manager.cc:4841] TS dd710cc670f54fa491a96609dcb96ce9 (127.18.159.130:38649): tablet af586d65708a451689cbce278ad0067d (table test-table [id=955a1089bd6e44bab0ca9e7e66139526]) successfully deleted
I20260926 02:07:04.187989 21413 tablet_bootstrap.cc:492] T 6ec64868df554c0c87c032a19c91d800 P 441d63ac043b43ce9e79d6a7f53c5434: No bootstrap required, opened a new log
I20260926 02:07:04.187989 21411 tablet_bootstrap.cc:492] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: No bootstrap required, opened a new log
I20260926 02:07:04.188210 21413 ts_tablet_manager.cc:1397] T 6ec64868df554c0c87c032a19c91d800 P 441d63ac043b43ce9e79d6a7f53c5434: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 02:07:04.188287 21411 ts_tablet_manager.cc:1397] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 02:07:04.189010 21411 raft_consensus.cc:348] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:04.189010 21413 raft_consensus.cc:348] T 6ec64868df554c0c87c032a19c91d800 P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:04.189275 21413 raft_consensus.cc:374] T 6ec64868df554c0c87c032a19c91d800 P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:04.189275 21411 raft_consensus.cc:374] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:04.189172 21412 tablet_bootstrap.cc:492] T 6ec64868df554c0c87c032a19c91d800 P dd710cc670f54fa491a96609dcb96ce9: No bootstrap required, opened a new log
I20260926 02:07:04.189357 21411 raft_consensus.cc:729] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d7d8d8cb3b343ee9ddb2a73a5eb3e43, State: Initialized, Role: FOLLOWER
I20260926 02:07:04.189357 21413 raft_consensus.cc:729] T 6ec64868df554c0c87c032a19c91d800 P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441d63ac043b43ce9e79d6a7f53c5434, State: Initialized, Role: FOLLOWER
I20260926 02:07:04.189632 21412 ts_tablet_manager.cc:1397] T 6ec64868df554c0c87c032a19c91d800 P dd710cc670f54fa491a96609dcb96ce9: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260926 02:07:04.189607 21411 consensus_queue.cc:260] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:04.190176 21411 ts_tablet_manager.cc:1428] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 02:07:04.190625 21412 raft_consensus.cc:348] T 6ec64868df554c0c87c032a19c91d800 P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:04.190953 21412 raft_consensus.cc:374] T 6ec64868df554c0c87c032a19c91d800 P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:04.191054 21412 raft_consensus.cc:729] T 6ec64868df554c0c87c032a19c91d800 P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd710cc670f54fa491a96609dcb96ce9, State: Initialized, Role: FOLLOWER
I20260926 02:07:04.191324 21412 consensus_queue.cc:260] T 6ec64868df554c0c87c032a19c91d800 P dd710cc670f54fa491a96609dcb96ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:04.191946 21412 ts_tablet_manager.cc:1428] T 6ec64868df554c0c87c032a19c91d800 P dd710cc670f54fa491a96609dcb96ce9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:07:04.192348 21413 consensus_queue.cc:260] T 6ec64868df554c0c87c032a19c91d800 P 441d63ac043b43ce9e79d6a7f53c5434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:04.192948 21413 ts_tablet_manager.cc:1428] T 6ec64868df554c0c87c032a19c91d800 P 441d63ac043b43ce9e79d6a7f53c5434: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20260926 02:07:04.214882 21395 raft_consensus.cc:482] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:07:04.215175 21395 raft_consensus.cc:504] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:04.216306 21395 leader_election.cc:288] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd710cc670f54fa491a96609dcb96ce9 (127.18.159.130:38649), 441d63ac043b43ce9e79d6a7f53c5434 (127.18.159.131:37133)
I20260926 02:07:04.216555 21121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec64868df554c0c87c032a19c91d800" candidate_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd710cc670f54fa491a96609dcb96ce9" is_pre_election: true
I20260926 02:07:04.216861 21121 raft_consensus.cc:2454] T 6ec64868df554c0c87c032a19c91d800 P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 in term 0.
I20260926 02:07:04.217003 21256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec64868df554c0c87c032a19c91d800" candidate_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441d63ac043b43ce9e79d6a7f53c5434" is_pre_election: true
I20260926 02:07:04.217267 21256 raft_consensus.cc:2454] T 6ec64868df554c0c87c032a19c91d800 P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 in term 0.
I20260926 02:07:04.217481 20907 leader_election.cc:302] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d7d8d8cb3b343ee9ddb2a73a5eb3e43, dd710cc670f54fa491a96609dcb96ce9; no voters:
I20260926 02:07:04.217774 21395 raft_consensus.cc:2781] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:07:04.217903 21395 raft_consensus.cc:482] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:07:04.217998 21395 raft_consensus.cc:3037] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:04.220078 21395 raft_consensus.cc:504] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:04.220708 21395 leader_election.cc:288] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [CANDIDATE]: Term 1 election: Requested vote from peers dd710cc670f54fa491a96609dcb96ce9 (127.18.159.130:38649), 441d63ac043b43ce9e79d6a7f53c5434 (127.18.159.131:37133)
I20260926 02:07:04.221257 21121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec64868df554c0c87c032a19c91d800" candidate_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd710cc670f54fa491a96609dcb96ce9"
I20260926 02:07:04.221251 21256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec64868df554c0c87c032a19c91d800" candidate_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441d63ac043b43ce9e79d6a7f53c5434"
I20260926 02:07:04.221508 21121 raft_consensus.cc:3037] T 6ec64868df554c0c87c032a19c91d800 P dd710cc670f54fa491a96609dcb96ce9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:04.221508 21256 raft_consensus.cc:3037] T 6ec64868df554c0c87c032a19c91d800 P 441d63ac043b43ce9e79d6a7f53c5434 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:04.224727 21256 raft_consensus.cc:2454] T 6ec64868df554c0c87c032a19c91d800 P 441d63ac043b43ce9e79d6a7f53c5434 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 in term 1.
I20260926 02:07:04.224858 21121 raft_consensus.cc:2454] T 6ec64868df554c0c87c032a19c91d800 P dd710cc670f54fa491a96609dcb96ce9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 in term 1.
I20260926 02:07:04.225356 20909 leader_election.cc:302] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 441d63ac043b43ce9e79d6a7f53c5434, 4d7d8d8cb3b343ee9ddb2a73a5eb3e43, dd710cc670f54fa491a96609dcb96ce9; no voters:
I20260926 02:07:04.225715 21395 raft_consensus.cc:2781] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:04.225955 21395 raft_consensus.cc:686] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 LEADER]: Becoming Leader. State: Replica: 4d7d8d8cb3b343ee9ddb2a73a5eb3e43, State: Running, Role: LEADER
I20260926 02:07:04.226287 21395 consensus_queue.cc:237] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } }
I20260926 02:07:04.229395 20819 catalog_manager.cc:5495] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 (127.18.159.129). New cstate: current_term: 1 leader_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d7d8d8cb3b343ee9ddb2a73a5eb3e43" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 } health_report { overall_health: UNKNOWN } } }
I20260926 02:07:04.261127 21420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:04.253679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39400 (local address 127.18.159.129:34257)
0926 02:07:04.253847 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:04.253855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:04.253877 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:07:04.254302 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:04.254309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:04.254431 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:04.254542 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:07:04.254998 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:04.256651 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:04.258718 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:04.259448 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:04.259599 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:04.260246 (+ 647us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:07:04.260396 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:07:04.260504 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:04.260668 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 02:07:04.260775 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 02:07:04.282266 21256 raft_consensus.cc:1261] T 6ec64868df554c0c87c032a19c91d800 P 441d63ac043b43ce9e79d6a7f53c5434 [term 1 FOLLOWER]: Refusing update from remote peer 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:04.283052 21121 raft_consensus.cc:1261] T 6ec64868df554c0c87c032a19c91d800 P dd710cc670f54fa491a96609dcb96ce9 [term 1 FOLLOWER]: Refusing update from remote peer 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:04.283061 21395 consensus_queue.cc:1035] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441d63ac043b43ce9e79d6a7f53c5434" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:04.283844 21332 consensus_queue.cc:1035] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd710cc670f54fa491a96609dcb96ce9" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 02:07:04.318698 21381 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 02:07:05.936299 20818 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:05.936542 20818 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:05.940814 20818 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e76c06d8fc9142adafcd660c2102fa24: Sending DeleteTablet for 3 replicas of tablet 6ec64868df554c0c87c032a19c91d800
I20260926 02:07:05.941696 20956 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ec64868df554c0c87c032a19c91d800 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:05 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:39358
I20260926 02:07:05.941740 21236 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ec64868df554c0c87c032a19c91d800 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:05 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:50622
I20260926 02:07:05.941946 21100 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ec64868df554c0c87c032a19c91d800 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:05 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:57446
I20260926 02:07:05.942947 21464 tablet_replica.cc:330] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: stopping tablet replica
I20260926 02:07:05.943017 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 20890
I20260926 02:07:05.943226 21464 raft_consensus.cc:2229] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:07:05.943604 21464 raft_consensus.cc:2258] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:05.945513 21464 ts_tablet_manager.cc:1905] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:05.955538 21464 ts_tablet_manager.cc:1918] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:07:05.955834 21464 log.cc:1196] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/6ec64868df554c0c87c032a19c91d800
I20260926 02:07:05.956923 21464 ts_tablet_manager.cc:1939] T 6ec64868df554c0c87c032a19c91d800 P 4d7d8d8cb3b343ee9ddb2a73a5eb3e43: Deleting consensus metadata
I20260926 02:07:05.957635 21465 tablet_replica.cc:330] T 6ec64868df554c0c87c032a19c91d800 P 441d63ac043b43ce9e79d6a7f53c5434: stopping tablet replica
I20260926 02:07:05.957927 21465 raft_consensus.cc:2229] T 6ec64868df554c0c87c032a19c91d800 P 441d63ac043b43ce9e79d6a7f53c5434 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:05.958263 21465 raft_consensus.cc:2258] T 6ec64868df554c0c87c032a19c91d800 P 441d63ac043b43ce9e79d6a7f53c5434 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:05.958263 20806 catalog_manager.cc:4841] TS 4d7d8d8cb3b343ee9ddb2a73a5eb3e43 (127.18.159.129:34257): tablet 6ec64868df554c0c87c032a19c91d800 (table test-table [id=60465baeb77a4bd1ba8cf96c7c73b8ff]) successfully deleted
I20260926 02:07:05.958952 21466 tablet_replica.cc:330] T 6ec64868df554c0c87c032a19c91d800 P dd710cc670f54fa491a96609dcb96ce9: stopping tablet replica
I20260926 02:07:05.959179 21466 raft_consensus.cc:2229] T 6ec64868df554c0c87c032a19c91d800 P dd710cc670f54fa491a96609dcb96ce9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:05.959406 21466 raft_consensus.cc:2258] T 6ec64868df554c0c87c032a19c91d800 P dd710cc670f54fa491a96609dcb96ce9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:05.960355 21465 ts_tablet_manager.cc:1905] T 6ec64868df554c0c87c032a19c91d800 P 441d63ac043b43ce9e79d6a7f53c5434: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:05.960417 21466 ts_tablet_manager.cc:1905] T 6ec64868df554c0c87c032a19c91d800 P dd710cc670f54fa491a96609dcb96ce9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:05.965961 21466 ts_tablet_manager.cc:1918] T 6ec64868df554c0c87c032a19c91d800 P dd710cc670f54fa491a96609dcb96ce9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:07:05.966135 21466 log.cc:1196] T 6ec64868df554c0c87c032a19c91d800 P dd710cc670f54fa491a96609dcb96ce9: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/6ec64868df554c0c87c032a19c91d800
I20260926 02:07:05.966529 21466 ts_tablet_manager.cc:1939] T 6ec64868df554c0c87c032a19c91d800 P dd710cc670f54fa491a96609dcb96ce9: Deleting consensus metadata
Sep 26 02:07:05 dist-test-slave-03rn krb5kdc[20774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388423, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:07:05.967516 21465 ts_tablet_manager.cc:1918] T 6ec64868df554c0c87c032a19c91d800 P 441d63ac043b43ce9e79d6a7f53c5434: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:07:05.967520 20804 catalog_manager.cc:4841] TS dd710cc670f54fa491a96609dcb96ce9 (127.18.159.130:38649): tablet 6ec64868df554c0c87c032a19c91d800 (table test-table [id=60465baeb77a4bd1ba8cf96c7c73b8ff]) successfully deleted
I20260926 02:07:05.967669 21465 log.cc:1196] T 6ec64868df554c0c87c032a19c91d800 P 441d63ac043b43ce9e79d6a7f53c5434: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/6ec64868df554c0c87c032a19c91d800
I20260926 02:07:05.968180 21465 ts_tablet_manager.cc:1939] T 6ec64868df554c0c87c032a19c91d800 P 441d63ac043b43ce9e79d6a7f53c5434: Deleting consensus metadata
I20260926 02:07:05.969214 20806 catalog_manager.cc:4841] TS 441d63ac043b43ce9e79d6a7f53c5434 (127.18.159.131:37133): tablet 6ec64868df554c0c87c032a19c91d800 (table test-table [id=60465baeb77a4bd1ba8cf96c7c73b8ff]) successfully deleted
I20260926 02:07:05.974707 21468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:05.944794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39416 (local address 127.18.159.129:34257)
0926 02:07:05.955645 (+ 10851us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:05.955654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:05.955684 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:07:05.955753 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:05.955758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:05.955856 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:05.955993 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:05.958143 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:05.959929 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:05.961963 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:05.962749 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:05.962883 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:05.968708 (+ 5825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:05.968724 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:05.968739 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:05.968781 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:05.971664 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:05.972538 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:05.972547 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:05.972550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:05.972703 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:05.973087 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:05.973091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:05.973093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:05.973502 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:05.973844 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:05.974112 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 02:07:05.974288 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10683,"thread_start_us":11353,"threads_started":1}
W20260926 02:07:05.975363 21021 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39416
W20260926 02:07:05.976049 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 20890: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:07:05.976361 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 20890
I20260926 02:07:06.001907 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 21030
Sep 26 02:07:06 dist-test-slave-03rn krb5kdc[20774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388423, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.159.130@KRBTEST.COM
I20260926 02:07:06.017809 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:06.002697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57456 (local address 127.18.159.130:38649)
0926 02:07:06.003280 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:06.003284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:06.003300 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:07:06.003353 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:06.003356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:06.003417 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:06.003506 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:06.004825 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:06.005599 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:06.006879 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:06.007322 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:06.007422 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:06.011887 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:06.011903 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:06.011919 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:06.011960 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:06.014857 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:06.015586 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:06.015594 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:06.015597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:06.015739 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:06.016134 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:06.016139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:06.016142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:06.016581 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:06.016909 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:06.017202 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 02:07:06.017444 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":365,"threads_started":1}
W20260926 02:07:06.018479 21161 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57456
W20260926 02:07:06.019207 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 21030: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:07:06.019363 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 21030
I20260926 02:07:06.041402 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 21170
Sep 26 02:07:06 dist-test-slave-03rn krb5kdc[20774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388423, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.159.131@KRBTEST.COM
I20260926 02:07:06.057842 21470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:06.042393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50630 (local address 127.18.159.131:37133)
0926 02:07:06.042952 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:06.042957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:06.042974 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:07:06.043030 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:06.043034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:06.043096 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:06.043195 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:06.044597 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:06.045493 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:06.046852 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:06.047325 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:06.047429 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:06.051686 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:06.051702 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:06.051718 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:06.051755 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:06.054858 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:06.055542 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:06.055550 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:06.055552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:06.055694 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:06.056023 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:06.056027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:06.056029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:06.056718 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:06.056963 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:06.057243 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 02:07:06.057436 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":304,"threads_started":1}
W20260926 02:07:06.058604 21301 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50630
W20260926 02:07:06.059303 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 21170: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:07:06.059484 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 21170
I20260926 02:07:06.077045 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 20787
I20260926 02:07:06.078347 20859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:06.193287 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 20787
2026-09-26T02:07:06Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (10384 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:07:06 dist-test-slave-03rn krb5kdc[21474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:07:06 dist-test-slave-03rn krb5kdc[21474](info): set up 2 sockets
Sep 26 02:07:06 dist-test-slave-03rn krb5kdc[21474](info): commencing operation
krb5kdc: starting...
W20260926 02:07:07.301520 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.006s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:07:07 dist-test-slave-03rn krb5kdc[21474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:07:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:07:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:07:07.507436 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:43167
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:43167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:07.850442 21487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:07.850757 21487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:07.850907 21487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:07.858273 21487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:07:07.858407 21487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:07.858456 21487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:07.858575 21487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:07:07.858659 21487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:07:07.868788 21487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:43167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:43167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:07.870167 21487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:07.871548 21487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:07.884507 21492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:07.885025 21493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:07.884996 21495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:07.886325 21487 server_base.cc:1034] running on GCE node
I20260926 02:07:07.887444 21487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:07.889180 21487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:07.890424 21487 hybrid_clock.cc:648] HybridClock initialized: now 1790388427890377 us; error 27 us; skew 500 ppm
I20260926 02:07:07.890770 21487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:07 dist-test-slave-03rn krb5kdc[21474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388427, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:07.894181 21487 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:07:07.895854 21487 webserver.cc:460] Webserver started at http://127.18.159.190:36017/ using document root <none> and password file <none>
I20260926 02:07:07.896672 21487 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:07.896803 21487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:07.897138 21487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:07.900308 21487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "fcbf594a12c24edeb1911860fd483133"
format_stamp: "Formatted at 2026-09-26 02:07:07 on dist-test-slave-03rn"
server_key: "1bee76d5049418182a4e6de16cfd8d38"
server_key_iv: "4abb37e054ccd4a89d567664b39605c5"
server_key_version: "encryptionkey@0"
I20260926 02:07:07.901151 21487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "fcbf594a12c24edeb1911860fd483133"
format_stamp: "Formatted at 2026-09-26 02:07:07 on dist-test-slave-03rn"
server_key: "1bee76d5049418182a4e6de16cfd8d38"
server_key_iv: "4abb37e054ccd4a89d567664b39605c5"
server_key_version: "encryptionkey@0"
I20260926 02:07:07.906724 21487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:07:07.910755 21502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:07.912250 21487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:07:07.912456 21487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "fcbf594a12c24edeb1911860fd483133"
format_stamp: "Formatted at 2026-09-26 02:07:07 on dist-test-slave-03rn"
server_key: "1bee76d5049418182a4e6de16cfd8d38"
server_key_iv: "4abb37e054ccd4a89d567664b39605c5"
server_key_version: "encryptionkey@0"
I20260926 02:07:07.912685 21487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:07.927659 21487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:07.932274 21487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:07.932607 21487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:07.951993 21487 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:43167
I20260926 02:07:07.952011 21553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:43167 every 8 connection(s)
I20260926 02:07:07.953555 21487 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:07:07.957692 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 21487
I20260926 02:07:07.957949 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:07:07.957949 21554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:07.958618 19070 external_mini_cluster.cc:1428] Setting key 31c45cff2ebe3232006447cb46d7a712
I20260926 02:07:07.973457 21554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133: Bootstrap starting.
Sep 26 02:07:07 dist-test-slave-03rn krb5kdc[21474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:07.977501 21554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:07.979036 21554 log.cc:824] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:07.983183 21554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133: No bootstrap required, opened a new log
I20260926 02:07:07.985085 21557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:07.960964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51190 (local address 127.18.159.190:43167)
0926 02:07:07.961887 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:07.961903 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:07.961970 (+ 67us) server_negotiation.cc:407] Connection header received
0926 02:07:07.963713 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:07.963752 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:07.964406 (+ 654us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:07.964938 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:07.966336 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:07.968310 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:07.969801 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:07.971051 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:07.971265 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:07.975523 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:07.975535 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:07.975557 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:07.975614 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:07.979630 (+ 4016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:07.980779 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:07.980790 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:07.980799 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:07.981033 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:07.981493 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:07.981497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:07.981500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:07.982312 (+ 812us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:07.982653 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:07.983103 (+ 450us) server_negotiation.cc:299] Negotiation successful
0926 02:07:07.983574 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":399,"threads_started":1}
I20260926 02:07:07.992448 21554 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcbf594a12c24edeb1911860fd483133" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 43167 } }
I20260926 02:07:07.992915 21554 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:07.992995 21554 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcbf594a12c24edeb1911860fd483133, State: Initialized, Role: FOLLOWER
I20260926 02:07:07.993680 21554 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcbf594a12c24edeb1911860fd483133" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 43167 } }
I20260926 02:07:07.993974 21554 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:07:07.994164 21554 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:07:07.994362 21554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:07.996829 21554 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcbf594a12c24edeb1911860fd483133" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 43167 } }
I20260926 02:07:07.997426 21554 leader_election.cc:302] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcbf594a12c24edeb1911860fd483133; no voters:
I20260926 02:07:07.998070 21554 leader_election.cc:288] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:07:07.998293 21559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:07.999847 21559 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133 [term 1 LEADER]: Becoming Leader. State: Replica: fcbf594a12c24edeb1911860fd483133, State: Running, Role: LEADER
I20260926 02:07:08.000588 21559 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcbf594a12c24edeb1911860fd483133" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 43167 } }
I20260926 02:07:08.000928 21554 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:07:08.004235 21560 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcbf594a12c24edeb1911860fd483133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcbf594a12c24edeb1911860fd483133" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 43167 } } }
I20260926 02:07:08.004760 21560 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133 [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:08.005244 21561 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcbf594a12c24edeb1911860fd483133. Latest consensus state: current_term: 1 leader_uuid: "fcbf594a12c24edeb1911860fd483133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcbf594a12c24edeb1911860fd483133" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 43167 } } }
I20260926 02:07:08.005607 21561 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133 [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:08.006551 21567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:07:08.011710 21567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:07:08.021353 21567 catalog_manager.cc:1347] Generated new cluster ID: 546e0ab052fb4e28893e845c034d31f9
I20260926 02:07:08.021497 21567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:07:08.058796 21567 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:07:08.060475 21567 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:07:08.081905 21567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcbf594a12c24edeb1911860fd483133: Generated new TSK 0
I20260926 02:07:08.082882 21567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.159.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
I20260926 02:07:08.131440 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:43167
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:08.477861 21580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:08.478178 21580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:08.478332 21580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:08.486219 21580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:08.486347 21580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:08.486515 21580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:07:08.496567 21580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:43167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:08.497884 21580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:08.499279 21580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:08.513356 21585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:08.513305 21588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:08.513275 21586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:08.514322 21580 server_base.cc:1034] running on GCE node
I20260926 02:07:08.515019 21580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:08.516067 21580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:08.517375 21580 hybrid_clock.cc:648] HybridClock initialized: now 1790388428517291 us; error 66 us; skew 500 ppm
I20260926 02:07:08.517756 21580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:08 dist-test-slave-03rn krb5kdc[21474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388428, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:08.521346 21580 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:07:08.523006 21580 webserver.cc:460] Webserver started at http://127.18.159.129:36561/ using document root <none> and password file <none>
I20260926 02:07:08.523718 21580 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:08.523823 21580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:08.524101 21580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:08.527262 21580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "deee009bafbf4a0f8e4404165949df95"
format_stamp: "Formatted at 2026-09-26 02:07:08 on dist-test-slave-03rn"
server_key: "7725848c65cb081e9a4c92104b174d08"
server_key_iv: "bc8b37d012c87950695434b37da34b23"
server_key_version: "encryptionkey@0"
I20260926 02:07:08.528259 21580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "deee009bafbf4a0f8e4404165949df95"
format_stamp: "Formatted at 2026-09-26 02:07:08 on dist-test-slave-03rn"
server_key: "7725848c65cb081e9a4c92104b174d08"
server_key_iv: "bc8b37d012c87950695434b37da34b23"
server_key_version: "encryptionkey@0"
I20260926 02:07:08.533962 21580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 02:07:08.537722 21595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:08.539318 21580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:07:08.539531 21580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "deee009bafbf4a0f8e4404165949df95"
format_stamp: "Formatted at 2026-09-26 02:07:08 on dist-test-slave-03rn"
server_key: "7725848c65cb081e9a4c92104b174d08"
server_key_iv: "bc8b37d012c87950695434b37da34b23"
server_key_version: "encryptionkey@0"
I20260926 02:07:08.539759 21580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:08.572676 21580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:08.577822 21580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:08.578140 21580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:08.579196 21580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:08.580691 21580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:08.580790 21580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:08.580943 21580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:08.581012 21580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:08.615214 21580 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:40913
I20260926 02:07:08.615260 21707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:40913 every 8 connection(s)
I20260926 02:07:08.616703 21580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:07:08.621627 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 21580
I20260926 02:07:08.621848 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:07:08.622597 19070 external_mini_cluster.cc:1428] Setting key 5d0faea64fe12234b066b83a613d6722
Sep 26 02:07:08 dist-test-slave-03rn krb5kdc[21474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388428, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:08.638105 21710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:08.619371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:43853 (local address 127.18.159.190:43167)
0926 02:07:08.619815 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:08.619821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:08.620902 (+ 1081us) server_negotiation.cc:407] Connection header received
0926 02:07:08.623097 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:08.623102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:08.623199 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:08.623346 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:08.625207 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:08.625904 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:08.627501 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:08.627911 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:08.628047 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:08.631973 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:08.631979 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:08.631982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:08.632018 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:08.634734 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:08.635584 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:08.635590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:08.635599 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:08.635723 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:08.636223 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:08.636233 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:08.636235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:08.636518 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:08.636682 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:08.637474 (+ 792us) server_negotiation.cc:299] Negotiation successful
0926 02:07:08.637720 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":207,"threads_started":1}
I20260926 02:07:08.639247 21711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:08.619969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:43167 (local address 127.18.159.129:43853)
0926 02:07:08.620741 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:08.620780 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:08.622778 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:08.623614 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:08.623645 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:08.624424 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:08.625034 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:08.625053 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:08.626072 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:08.626076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:08.627345 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:08.627358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:08.628146 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:08.628180 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:08.628424 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:08.629193 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:08.629223 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:08.631692 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:08.634924 (+ 3232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:08.634935 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:08.634957 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:08.635437 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:08.635842 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:08.635847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:08.635851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:08.636064 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:08.636661 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:08.636669 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:08.637071 (+ 402us) client_negotiation.cc:769] Sending connection context
0926 02:07:08.637367 (+ 296us) client_negotiation.cc:240] Negotiation successful
0926 02:07:08.637970 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":361,"threads_started":1}
WARNING: no policy specified for kudu/127.18.159.130@KRBTEST.COM; defaulting to no policy
I20260926 02:07:08.641222 21708 heartbeater.cc:347] Connected to a master server at 127.18.159.190:43167
I20260926 02:07:08.641587 21708 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:08.642413 21708 heartbeater.cc:511] Master 127.18.159.190:43167 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.130@KRBTEST.COM" created.
I20260926 02:07:08.645202 21519 ts_manager.cc:194] Registered new tserver with Master: deee009bafbf4a0f8e4404165949df95 (127.18.159.129:40913)
I20260926 02:07:08.647483 21519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:43853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
I20260926 02:07:08.669535 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:43167
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:09.012688 21714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:09.013003 21714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:09.013099 21714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:09.020989 21714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:09.021113 21714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:09.021234 21714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:07:09.031248 21714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:43167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:09.032675 21714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:09.034018 21714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:09.048768 21720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:09.048856 21722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:09.049149 21719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:09.049763 21714 server_base.cc:1034] running on GCE node
I20260926 02:07:09.050408 21714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:09.051383 21714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:09.052625 21714 hybrid_clock.cc:648] HybridClock initialized: now 1790388429052569 us; error 62 us; skew 500 ppm
I20260926 02:07:09.052968 21714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:09 dist-test-slave-03rn krb5kdc[21474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388429, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:09.060026 21714 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:07:09.061766 21714 webserver.cc:460] Webserver started at http://127.18.159.130:40229/ using document root <none> and password file <none>
I20260926 02:07:09.062503 21714 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:09.062618 21714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:09.062888 21714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:09.066012 21714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "7af46da758bf4c0ab19d6af4e33f5f8c"
format_stamp: "Formatted at 2026-09-26 02:07:09 on dist-test-slave-03rn"
server_key: "9f9f023354f41a2b4fcf7dcd4e1168a3"
server_key_iv: "74f7aa7e16ac7c9c61e86ca59bf51672"
server_key_version: "encryptionkey@0"
I20260926 02:07:09.066823 21714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "7af46da758bf4c0ab19d6af4e33f5f8c"
format_stamp: "Formatted at 2026-09-26 02:07:09 on dist-test-slave-03rn"
server_key: "9f9f023354f41a2b4fcf7dcd4e1168a3"
server_key_iv: "74f7aa7e16ac7c9c61e86ca59bf51672"
server_key_version: "encryptionkey@0"
I20260926 02:07:09.072757 21714 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 02:07:09.076870 21729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:09.078583 21714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:07:09.078775 21714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "7af46da758bf4c0ab19d6af4e33f5f8c"
format_stamp: "Formatted at 2026-09-26 02:07:09 on dist-test-slave-03rn"
server_key: "9f9f023354f41a2b4fcf7dcd4e1168a3"
server_key_iv: "74f7aa7e16ac7c9c61e86ca59bf51672"
server_key_version: "encryptionkey@0"
I20260926 02:07:09.079016 21714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:09.104592 21714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:09.109473 21714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:09.109788 21714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:09.110864 21714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:09.112304 21714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:09.112380 21714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:09.112521 21714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:09.112569 21714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:09.146826 21714 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:46731
I20260926 02:07:09.146946 21841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:46731 every 8 connection(s)
I20260926 02:07:09.148177 21714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:07:09.150748 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 21714
I20260926 02:07:09.150965 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:07:09.151459 19070 external_mini_cluster.cc:1428] Setting key b5b528197ede300165e557e7643b4289
Sep 26 02:07:09 dist-test-slave-03rn krb5kdc[21474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388429, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.159.131@KRBTEST.COM; defaulting to no policy
I20260926 02:07:09.171756 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:09.150804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:47207 (local address 127.18.159.190:43167)
0926 02:07:09.151301 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:09.151308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:09.152508 (+ 1200us) server_negotiation.cc:407] Connection header received
0926 02:07:09.155111 (+ 2603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:09.155116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:09.155203 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:09.155306 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:09.157825 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:09.158612 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:09.161265 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:09.161724 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:09.161873 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:09.165693 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:09.165699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:09.165702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:09.165738 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:09.168525 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:09.169340 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:09.169344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:09.169347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:09.169471 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:09.169928 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:09.169932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:09.169934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:09.170195 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:09.170345 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:09.171134 (+ 789us) server_negotiation.cc:299] Negotiation successful
0926 02:07:09.171361 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":254,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.131@KRBTEST.COM" created.
I20260926 02:07:09.172848 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:09.151408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:43167 (local address 127.18.159.130:47207)
0926 02:07:09.152310 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:09.152366 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:09.154718 (+ 2352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:09.155648 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:09.155664 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:09.156599 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:09.157566 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:09.157600 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:09.159083 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:09.159093 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:09.161050 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:09.161068 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:09.162002 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:09.162010 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:09.162268 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:09.163016 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:09.163042 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:09.165378 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:09.168692 (+ 3314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:09.168703 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:09.168724 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:09.169195 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:09.169584 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:09.169590 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:09.169594 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:09.169795 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:09.170329 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:09.170337 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:09.170776 (+ 439us) client_negotiation.cc:769] Sending connection context
0926 02:07:09.171040 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 02:07:09.171624 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":300,"threads_started":1}
I20260926 02:07:09.174654 21842 heartbeater.cc:347] Connected to a master server at 127.18.159.190:43167
I20260926 02:07:09.174985 21842 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:09.175734 21842 heartbeater.cc:511] Master 127.18.159.190:43167 requested a full tablet report, sending...
I20260926 02:07:09.177435 21519 ts_manager.cc:194] Registered new tserver with Master: 7af46da758bf4c0ab19d6af4e33f5f8c (127.18.159.130:46731)
I20260926 02:07:09.178460 21519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:47207
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
I20260926 02:07:09.199124 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:43167
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:09.550773 21848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:09.551105 21848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:09.551275 21848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:09.559041 21848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:09.559176 21848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:09.559362 21848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:07:09.569783 21848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:43167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:09.571203 21848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:09.572729 21848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:09.587172 21856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:09.587251 21854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:09.587224 21853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:09.588717 21848 server_base.cc:1034] running on GCE node
I20260926 02:07:09.589387 21848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:09.590566 21848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:09.591879 21848 hybrid_clock.cc:648] HybridClock initialized: now 1790388429591778 us; error 94 us; skew 500 ppm
I20260926 02:07:09.592381 21848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:09 dist-test-slave-03rn krb5kdc[21474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388429, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:09.595774 21848 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:07:09.597615 21848 webserver.cc:460] Webserver started at http://127.18.159.131:45793/ using document root <none> and password file <none>
I20260926 02:07:09.598489 21848 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:09.598647 21848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:09.599007 21848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:09.602763 21848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "afc6aa2539644b038bf22b64dece3ebe"
format_stamp: "Formatted at 2026-09-26 02:07:09 on dist-test-slave-03rn"
server_key: "db3a2942415c1a80d1503c785916012e"
server_key_iv: "a4cd0e1ac7e0aecb7dd546cf27efcdad"
server_key_version: "encryptionkey@0"
I20260926 02:07:09.603706 21848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "afc6aa2539644b038bf22b64dece3ebe"
format_stamp: "Formatted at 2026-09-26 02:07:09 on dist-test-slave-03rn"
server_key: "db3a2942415c1a80d1503c785916012e"
server_key_iv: "a4cd0e1ac7e0aecb7dd546cf27efcdad"
server_key_version: "encryptionkey@0"
I20260926 02:07:09.609947 21848 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 02:07:09.614048 21863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:09.616012 21848 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 02:07:09.616250 21848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "afc6aa2539644b038bf22b64dece3ebe"
format_stamp: "Formatted at 2026-09-26 02:07:09 on dist-test-slave-03rn"
server_key: "db3a2942415c1a80d1503c785916012e"
server_key_iv: "a4cd0e1ac7e0aecb7dd546cf27efcdad"
server_key_version: "encryptionkey@0"
I20260926 02:07:09.616520 21848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:09.651659 21708 heartbeater.cc:503] Master 127.18.159.190:43167 was elected leader, sending a full tablet report...
I20260926 02:07:09.653662 21848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:09.658726 21848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:09.659162 21848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:09.660293 21848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:09.662089 21848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:09.662186 21848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:09.662349 21848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:09.662406 21848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:09.699005 21848 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:37811
I20260926 02:07:09.699056 21975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:37811 every 8 connection(s)
I20260926 02:07:09.700497 21848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:07:09.700937 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 21848
I20260926 02:07:09.701186 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:07:09.701836 19070 external_mini_cluster.cc:1428] Setting key f11003686b7630aafb7a1652733c2b04
Sep 26 02:07:09 dist-test-slave-03rn krb5kdc[21474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388429, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:09.725142 21978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:09.704048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:35233 (local address 127.18.159.190:43167)
0926 02:07:09.704585 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:09.704591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:09.706151 (+ 1560us) server_negotiation.cc:407] Connection header received
0926 02:07:09.708525 (+ 2374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:09.708531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:09.708631 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:09.708919 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:09.710739 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:09.711520 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:09.713323 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:09.713738 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:09.713885 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:09.718041 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:09.718048 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:09.718052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:09.718109 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:09.721445 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:09.722502 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:09.722508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:09.722511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:09.722711 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:09.723324 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:09.723328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:09.723330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:09.723622 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:09.723740 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:09.724665 (+ 925us) server_negotiation.cc:299] Negotiation successful
0926 02:07:09.724853 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":296,"threads_started":1}
I20260926 02:07:09.726282 21979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:09.704816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:43167 (local address 127.18.159.131:35233)
0926 02:07:09.705762 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:09.705814 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:09.708138 (+ 2324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:09.709214 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:09.709229 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:09.709855 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:09.710502 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:09.710522 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:09.711707 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:09.711711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:09.713152 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:09.713166 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:09.714005 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:09.714012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:09.714264 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:09.714987 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:09.715016 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:09.717753 (+ 2737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:09.721666 (+ 3913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:09.721676 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:09.721697 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:09.722339 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:09.722851 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:09.722856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:09.722859 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:09.723181 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:09.723762 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:09.723769 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:09.724214 (+ 445us) client_negotiation.cc:769] Sending connection context
0926 02:07:09.724522 (+ 308us) client_negotiation.cc:240] Negotiation successful
0926 02:07:09.725035 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":451,"threads_started":1}
I20260926 02:07:09.728056 21976 heartbeater.cc:347] Connected to a master server at 127.18.159.190:43167
I20260926 02:07:09.728475 21976 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:09.729310 21976 heartbeater.cc:511] Master 127.18.159.190:43167 requested a full tablet report, sending...
I20260926 02:07:09.730963 21518 ts_manager.cc:194] Registered new tserver with Master: afc6aa2539644b038bf22b64dece3ebe (127.18.159.131:37811)
I20260926 02:07:09.732033 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:07:09.732290 21518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:35233
W20260926 02:07:09.767068 21978 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:09.763902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51318 (local address 127.18.159.190:43167)
0926 02:07:09.764137 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:09.764148 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:09.764650 (+ 502us) server_negotiation.cc:407] Connection header received
0926 02:07:09.764812 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:09.764826 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:09.764895 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:09.765035 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:09.766781 (+ 1746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51318: BlockingRecv error: recv got EOF from 127.0.0.1:51318 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:07:09.770288 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 21580
W20260926 02:07:09.773411 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 21580: Not authorized: Client connection negotiation failed: client connection to 127.18.159.129:40913: server requires authentication, but client does not have Kerberos credentials available
I20260926 02:07:09.773675 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 21580
I20260926 02:07:09.790108 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 21714
W20260926 02:07:09.792863 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 21714: Not authorized: Client connection negotiation failed: client connection to 127.18.159.130:46731: server requires authentication, but client does not have Kerberos credentials available
I20260926 02:07:09.793049 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 21714
W20260926 02:07:09.793157 21989 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:09.791047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.18.159.130:46731)
0926 02:07:09.791630 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:09.791637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:09.791665 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:07:09.791726 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:09.791731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:09.791936 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:09.792072 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:09.792833 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50384: BlockingRecv error: recv got EOF from 127.0.0.1:50384 (error 108)
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":333,"threads_started":1}
I20260926 02:07:09.809122 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 21848
W20260926 02:07:09.813162 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 21848: Not authorized: Client connection negotiation failed: client connection to 127.18.159.131:37811: server requires authentication, but client does not have Kerberos credentials available
I20260926 02:07:09.813328 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 21848
I20260926 02:07:09.829722 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 21487
I20260926 02:07:09.831184 21549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:09.939219 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 21487
2026-09-26T02:07:09Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3743 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:07:10 dist-test-slave-03rn krb5kdc[21994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:07:10 dist-test-slave-03rn krb5kdc[21994](info): set up 2 sockets
Sep 26 02:07:10 dist-test-slave-03rn krb5kdc[21994](info): commencing operation
krb5kdc: starting...
W20260926 02:07:11.057037 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:07:11 dist-test-slave-03rn krb5kdc[21994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:07:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:07:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:07:11.264779 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:42595
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:41983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:42595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:11.598078 22007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:11.598392 22007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:11.598546 22007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:11.606009 22007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:07:11.606135 22007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:11.606181 22007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:11.606288 22007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:07:11.606362 22007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:07:11.616544 22007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:41983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:42595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:42595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:11.617890 22007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:11.619235 22007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:11.630867 22015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:11.630959 22012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:11.630955 22013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:11.632265 22007 server_base.cc:1034] running on GCE node
I20260926 02:07:11.633122 22007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:11.634644 22007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:11.635874 22007 hybrid_clock.cc:648] HybridClock initialized: now 1790388431635813 us; error 57 us; skew 500 ppm
I20260926 02:07:11.636332 22007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:11 dist-test-slave-03rn krb5kdc[21994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388431, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:11.639760 22007 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:07:11.641491 22007 webserver.cc:460] Webserver started at http://127.18.159.190:35645/ using document root <none> and password file <none>
I20260926 02:07:11.642292 22007 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:11.642429 22007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:11.642769 22007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:11.646136 22007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "e61c6594d6cc4198b39637ad9af60599"
format_stamp: "Formatted at 2026-09-26 02:07:11 on dist-test-slave-03rn"
server_key: "7c8acf81f7fe80585b93b0ef60762f22"
server_key_iv: "d7a1b542f34bd4e59591f7261e38f400"
server_key_version: "encryptionkey@0"
I20260926 02:07:11.647008 22007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "e61c6594d6cc4198b39637ad9af60599"
format_stamp: "Formatted at 2026-09-26 02:07:11 on dist-test-slave-03rn"
server_key: "7c8acf81f7fe80585b93b0ef60762f22"
server_key_iv: "d7a1b542f34bd4e59591f7261e38f400"
server_key_version: "encryptionkey@0"
I20260926 02:07:11.652663 22007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 02:07:11.656544 22022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:11.658083 22007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:07:11.658301 22007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "e61c6594d6cc4198b39637ad9af60599"
format_stamp: "Formatted at 2026-09-26 02:07:11 on dist-test-slave-03rn"
server_key: "7c8acf81f7fe80585b93b0ef60762f22"
server_key_iv: "d7a1b542f34bd4e59591f7261e38f400"
server_key_version: "encryptionkey@0"
I20260926 02:07:11.658519 22007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:11.672906 22007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:11.677625 22007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:11.677927 22007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:11.697292 22007 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:42595
I20260926 02:07:11.697315 22073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:42595 every 8 connection(s)
I20260926 02:07:11.698745 22007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:07:11.702772 22074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:11.704708 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 22007
I20260926 02:07:11.704995 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:07:11.705823 19070 external_mini_cluster.cc:1428] Setting key 56a0e5abddd4aa7271b99ac54a5c0508
I20260926 02:07:11.717782 22074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599: Bootstrap starting.
Sep 26 02:07:11 dist-test-slave-03rn krb5kdc[21994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:11.721419 22074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:11.722684 22074 log.cc:824] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:11.725912 22074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599: No bootstrap required, opened a new log
I20260926 02:07:11.729938 22077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:11.707765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35288 (local address 127.18.159.190:42595)
0926 02:07:11.708641 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:11.708653 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:11.708700 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:07:11.710258 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:11.710286 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:11.710711 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:11.711044 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:11.713081 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:11.714463 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:11.716331 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:11.717077 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:11.717227 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:11.721741 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:11.721750 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:11.721768 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:11.721817 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:11.725588 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:11.726438 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:11.726451 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:11.726466 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:11.726639 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:11.727028 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:11.727033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:11.727036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:11.727611 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:11.727939 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:11.728444 (+ 505us) server_negotiation.cc:299] Negotiation successful
0926 02:07:11.728882 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":447,"threads_started":1}
I20260926 02:07:11.734151 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:07:11.734512 22074 raft_consensus.cc:348] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61c6594d6cc4198b39637ad9af60599" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 42595 } }
I20260926 02:07:11.734890 22074 raft_consensus.cc:374] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:11.734994 22074 raft_consensus.cc:729] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e61c6594d6cc4198b39637ad9af60599, State: Initialized, Role: FOLLOWER
I20260926 02:07:11.735527 22074 consensus_queue.cc:260] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61c6594d6cc4198b39637ad9af60599" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 42595 } }
I20260926 02:07:11.735752 22074 raft_consensus.cc:388] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:07:11.735881 22074 raft_consensus.cc:482] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:07:11.736045 22074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:11.738256 22074 raft_consensus.cc:504] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61c6594d6cc4198b39637ad9af60599" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 42595 } }
I20260926 02:07:11.738837 22074 leader_election.cc:302] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e61c6594d6cc4198b39637ad9af60599; no voters:
I20260926 02:07:11.739559 22074 leader_election.cc:288] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:07:11.739656 22079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:11.742381 22074 sys_catalog.cc:564] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:07:11.742820 22079 raft_consensus.cc:686] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599 [term 1 LEADER]: Becoming Leader. State: Replica: e61c6594d6cc4198b39637ad9af60599, State: Running, Role: LEADER
I20260926 02:07:11.743494 22079 consensus_queue.cc:237] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61c6594d6cc4198b39637ad9af60599" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 42595 } }
I20260926 02:07:11.746856 22082 sys_catalog.cc:455] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e61c6594d6cc4198b39637ad9af60599. Latest consensus state: current_term: 1 leader_uuid: "e61c6594d6cc4198b39637ad9af60599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61c6594d6cc4198b39637ad9af60599" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 42595 } } }
I20260926 02:07:11.746955 22080 sys_catalog.cc:455] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e61c6594d6cc4198b39637ad9af60599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61c6594d6cc4198b39637ad9af60599" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 42595 } } }
I20260926 02:07:11.747328 22082 sys_catalog.cc:458] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599 [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:11.747336 22080 sys_catalog.cc:458] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599 [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:11.748366 22090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:07:11.766703 22090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:07:11.776610 22090 catalog_manager.cc:1347] Generated new cluster ID: 4d486740f433491e9b4d047d4a537d00
I20260926 02:07:11.776736 22090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:07:11.791782 22077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:11.782639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.18.159.190:42595)
0926 02:07:11.782806 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:11.782812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:11.783356 (+ 544us) server_negotiation.cc:407] Connection header received
0926 02:07:11.783534 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:11.783539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:11.783624 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:11.783737 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:11.784936 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:11.786133 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:11.787892 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:11.788579 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:11.788716 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:11.789900 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:11.789908 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:11.789919 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:07:11.790937 (+ 1018us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:11.791033 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:07:11.791075 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:11.791188 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:11.791314 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 02:07:11.791428 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:07:11.807024 22090 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:07:11.809149 22090 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:07:11.817157 22090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e61c6594d6cc4198b39637ad9af60599: Generated new TSK 0
I20260926 02:07:11.818217 22090 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:07:11.827702 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 22007
I20260926 02:07:11.829012 22069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:11.923098 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 22007
2026-09-26T02:07:11Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1976 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:07:12 dist-test-slave-03rn krb5kdc[22108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:07:12 dist-test-slave-03rn krb5kdc[22108](info): set up 2 sockets
Sep 26 02:07:12 dist-test-slave-03rn krb5kdc[22108](info): commencing operation
krb5kdc: starting...
W20260926 02:07:13.009068 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:07:13 dist-test-slave-03rn krb5kdc[22108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:07:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:07:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:07:13.212428 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:41369
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:41369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:13.553455 22121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:13.553768 22121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:13.553915 22121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:13.561489 22121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:07:13.561618 22121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:13.561709 22121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:13.561811 22121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:07:13.561872 22121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:07:13.572541 22121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:41369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:41369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:13.573938 22121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:13.575285 22121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:13.587568 22127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:13.587654 22126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:13.587488 22129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:13.588850 22121 server_base.cc:1034] running on GCE node
I20260926 02:07:13.589725 22121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:13.591368 22121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:13.592617 22121 hybrid_clock.cc:648] HybridClock initialized: now 1790388433592564 us; error 50 us; skew 500 ppm
I20260926 02:07:13.593004 22121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:13 dist-test-slave-03rn krb5kdc[22108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388433, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:13.596366 22121 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:07:13.597963 22121 webserver.cc:460] Webserver started at http://127.18.159.190:45103/ using document root <none> and password file <none>
I20260926 02:07:13.598646 22121 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:13.598752 22121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:13.599050 22121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:13.602281 22121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "1c0c593e238445c68e89470a522c949d"
format_stamp: "Formatted at 2026-09-26 02:07:13 on dist-test-slave-03rn"
server_key: "e47fd926dec432f75cdc19c606890dec"
server_key_iv: "ad04cccf09844e4026464bd518d89fdc"
server_key_version: "encryptionkey@0"
I20260926 02:07:13.603097 22121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "1c0c593e238445c68e89470a522c949d"
format_stamp: "Formatted at 2026-09-26 02:07:13 on dist-test-slave-03rn"
server_key: "e47fd926dec432f75cdc19c606890dec"
server_key_iv: "ad04cccf09844e4026464bd518d89fdc"
server_key_version: "encryptionkey@0"
I20260926 02:07:13.608788 22121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 02:07:13.612483 22136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:13.613983 22121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:07:13.614172 22121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "1c0c593e238445c68e89470a522c949d"
format_stamp: "Formatted at 2026-09-26 02:07:13 on dist-test-slave-03rn"
server_key: "e47fd926dec432f75cdc19c606890dec"
server_key_iv: "ad04cccf09844e4026464bd518d89fdc"
server_key_version: "encryptionkey@0"
I20260926 02:07:13.614348 22121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:13.630971 22121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:13.635686 22121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:13.636003 22121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:13.655387 22121 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:41369
I20260926 02:07:13.655404 22187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:41369 every 8 connection(s)
I20260926 02:07:13.656911 22121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:07:13.661049 22188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:13.662962 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 22121
I20260926 02:07:13.663214 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:07:13.663879 19070 external_mini_cluster.cc:1428] Setting key ce55f30cf4ee18dd76f633ec2ca327c6
I20260926 02:07:13.673990 22188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d: Bootstrap starting.
I20260926 02:07:13.678220 22188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d: Neither blocks nor log segments found. Creating new log.
Sep 26 02:07:13 dist-test-slave-03rn krb5kdc[22108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:13.679589 22188 log.cc:824] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:13.683938 22188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d: No bootstrap required, opened a new log
I20260926 02:07:13.690886 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:13.666125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.18.159.190:41369)
0926 02:07:13.667007 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:13.667019 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:13.667063 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:07:13.668821 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:13.668847 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:13.669283 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:13.669653 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:13.671455 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:13.672964 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:13.674860 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:13.675606 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:13.675759 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:13.681156 (+ 5397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:13.681166 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:13.681185 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:13.681268 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:13.685580 (+ 4312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:13.686508 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:13.686534 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:13.686545 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:13.686777 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:13.687185 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:13.687190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:13.687193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:13.687986 (+ 793us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:13.688340 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:13.688824 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 02:07:13.689322 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":416,"threads_started":1}
I20260926 02:07:13.694082 22188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0c593e238445c68e89470a522c949d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 41369 } }
I20260926 02:07:13.694515 22188 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:13.694645 22188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0c593e238445c68e89470a522c949d, State: Initialized, Role: FOLLOWER
I20260926 02:07:13.695292 22188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0c593e238445c68e89470a522c949d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 41369 } }
I20260926 02:07:13.695729 22188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:07:13.695909 22188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:07:13.696107 22188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:13.698421 22188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0c593e238445c68e89470a522c949d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 41369 } }
I20260926 02:07:13.699004 22188 leader_election.cc:302] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c0c593e238445c68e89470a522c949d; no voters:
I20260926 02:07:13.699609 22188 leader_election.cc:288] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:07:13.699743 22193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:13.700497 22193 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d [term 1 LEADER]: Becoming Leader. State: Replica: 1c0c593e238445c68e89470a522c949d, State: Running, Role: LEADER
I20260926 02:07:13.701020 22193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0c593e238445c68e89470a522c949d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 41369 } }
I20260926 02:07:13.701952 22188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:07:13.704231 22195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c0c593e238445c68e89470a522c949d. Latest consensus state: current_term: 1 leader_uuid: "1c0c593e238445c68e89470a522c949d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0c593e238445c68e89470a522c949d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 41369 } } }
I20260926 02:07:13.704697 22195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:13.707074 22194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c0c593e238445c68e89470a522c949d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0c593e238445c68e89470a522c949d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 41369 } } }
I20260926 02:07:13.707461 22194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:13.707487 22202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:07:13.708791 22209 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 02:07:13.712798 22202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:07:13.722874 22202 catalog_manager.cc:1347] Generated new cluster ID: 604cd4de34e94378bab879f2bb07b294
I20260926 02:07:13.723024 22202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:07:13.737052 22202 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:07:13.739006 22202 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:07:13.751765 22202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c0c593e238445c68e89470a522c949d: Generated new TSK 0
I20260926 02:07:13.753024 22202 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.159.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
I20260926 02:07:13.811971 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:41369
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:14.162283 22214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:14.162601 22214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:14.162760 22214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:14.170281 22214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:14.170418 22214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:14.170599 22214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:07:14.180483 22214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:41369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:14.181883 22214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:14.183449 22214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:14.197506 22222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:14.197582 22220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:14.197540 22219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:14.198382 22214 server_base.cc:1034] running on GCE node
I20260926 02:07:14.199120 22214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:14.200271 22214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:14.201545 22214 hybrid_clock.cc:648] HybridClock initialized: now 1790388434201477 us; error 57 us; skew 500 ppm
I20260926 02:07:14.202042 22214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:14 dist-test-slave-03rn krb5kdc[22108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388434, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:14.211781 22214 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:07:14.213558 22214 webserver.cc:460] Webserver started at http://127.18.159.129:33195/ using document root <none> and password file <none>
I20260926 02:07:14.214308 22214 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:14.214459 22214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:14.214781 22214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:14.217854 22214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "816dcf12780744e8a81b0b4cbd1e9ecc"
format_stamp: "Formatted at 2026-09-26 02:07:14 on dist-test-slave-03rn"
server_key: "2597bac58303b73700fbed093d73716b"
server_key_iv: "09934bfedf0571673fd79c43a35d46ec"
server_key_version: "encryptionkey@0"
I20260926 02:07:14.218770 22214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "816dcf12780744e8a81b0b4cbd1e9ecc"
format_stamp: "Formatted at 2026-09-26 02:07:14 on dist-test-slave-03rn"
server_key: "2597bac58303b73700fbed093d73716b"
server_key_iv: "09934bfedf0571673fd79c43a35d46ec"
server_key_version: "encryptionkey@0"
I20260926 02:07:14.224612 22214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:07:14.228473 22229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:14.230221 22214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:07:14.230397 22214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "816dcf12780744e8a81b0b4cbd1e9ecc"
format_stamp: "Formatted at 2026-09-26 02:07:14 on dist-test-slave-03rn"
server_key: "2597bac58303b73700fbed093d73716b"
server_key_iv: "09934bfedf0571673fd79c43a35d46ec"
server_key_version: "encryptionkey@0"
I20260926 02:07:14.230597 22214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:14.251348 22214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:14.256367 22214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:14.256652 22214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:14.257640 22214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:14.259166 22214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:14.259240 22214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:14.259332 22214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:14.259375 22214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:14.294843 22214 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:45257
I20260926 02:07:14.294937 22341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:45257 every 8 connection(s)
I20260926 02:07:14.296332 22214 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:07:14.303366 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 22214
I20260926 02:07:14.303578 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:07:14.304245 19070 external_mini_cluster.cc:1428] Setting key 0fbd90efa9299d1d2ad1c72317595b41
Sep 26 02:07:14 dist-test-slave-03rn krb5kdc[22108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388434, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:14.319581 22344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:14.299122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:35825 (local address 127.18.159.190:41369)
0926 02:07:14.299535 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:14.299540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:14.300743 (+ 1203us) server_negotiation.cc:407] Connection header received
0926 02:07:14.302695 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:14.302701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:14.302806 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:14.302938 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:14.305127 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:14.305999 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:14.307806 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:14.308569 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:14.308716 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:14.312854 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:14.312860 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:14.312863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:14.312899 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:14.315550 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:14.316696 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:14.316701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:14.316704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:14.316839 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:14.317411 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:14.317415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:14.317417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:14.317762 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:14.317879 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:14.318986 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0926 02:07:14.319224 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":199,"threads_started":1}
I20260926 02:07:14.321226 22345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:14.299786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:41369 (local address 127.18.159.129:35825)
0926 02:07:14.300573 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:14.300618 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:14.302409 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:14.303327 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:14.303342 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:14.304090 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:14.304910 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:14.304950 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:14.306191 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:14.306199 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:14.307627 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:14.307641 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:14.308824 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:14.308830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:14.309046 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:14.309781 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:14.309809 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:14.312545 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:14.315732 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:14.315745 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:14.315768 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:14.316514 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:14.316965 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:14.316973 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:14.316979 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:14.317266 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:14.317899 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:14.317911 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:14.318471 (+ 560us) client_negotiation.cc:769] Sending connection context
0926 02:07:14.318824 (+ 353us) client_negotiation.cc:240] Negotiation successful
0926 02:07:14.319492 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":355,"threads_started":1}
I20260926 02:07:14.323036 22342 heartbeater.cc:347] Connected to a master server at 127.18.159.190:41369
I20260926 02:07:14.323424 22342 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.18.159.130@KRBTEST.COM; defaulting to no policy
I20260926 02:07:14.324447 22342 heartbeater.cc:511] Master 127.18.159.190:41369 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.130@KRBTEST.COM" created.
I20260926 02:07:14.326990 22153 ts_manager.cc:194] Registered new tserver with Master: 816dcf12780744e8a81b0b4cbd1e9ecc (127.18.159.129:45257)
I20260926 02:07:14.329312 22153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:35825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
I20260926 02:07:14.351969 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:41369
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:14.694578 22348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:14.694900 22348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:14.695060 22348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:14.702826 22348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:14.702967 22348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:14.703151 22348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:07:14.713598 22348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:41369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:14.714949 22348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:14.716323 22348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:14.732781 22354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:14.733732 22353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:14.734014 22348 server_base.cc:1034] running on GCE node
W20260926 02:07:14.736420 22356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:14.737421 22348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:14.739099 22348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:14.740501 22348 hybrid_clock.cc:648] HybridClock initialized: now 1790388434740407 us; error 77 us; skew 500 ppm
I20260926 02:07:14.740988 22348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:14 dist-test-slave-03rn krb5kdc[22108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388434, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:14.745792 22348 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:07:14.747323 22348 webserver.cc:460] Webserver started at http://127.18.159.130:35391/ using document root <none> and password file <none>
I20260926 02:07:14.748024 22348 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:14.748131 22348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:14.748600 22348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:14.753436 22348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "9b88486ad00c4e9d8d5583edfab8be26"
format_stamp: "Formatted at 2026-09-26 02:07:14 on dist-test-slave-03rn"
server_key: "f05ccab37ce21e456feaf84747cb73b5"
server_key_iv: "34ab64edcf22a05690d1d7f67c0ecb64"
server_key_version: "encryptionkey@0"
I20260926 02:07:14.754617 22348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "9b88486ad00c4e9d8d5583edfab8be26"
format_stamp: "Formatted at 2026-09-26 02:07:14 on dist-test-slave-03rn"
server_key: "f05ccab37ce21e456feaf84747cb73b5"
server_key_iv: "34ab64edcf22a05690d1d7f67c0ecb64"
server_key_version: "encryptionkey@0"
I20260926 02:07:14.762001 22348 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260926 02:07:14.767788 22363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:14.770006 22348 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260926 02:07:14.770268 22348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "9b88486ad00c4e9d8d5583edfab8be26"
format_stamp: "Formatted at 2026-09-26 02:07:14 on dist-test-slave-03rn"
server_key: "f05ccab37ce21e456feaf84747cb73b5"
server_key_iv: "34ab64edcf22a05690d1d7f67c0ecb64"
server_key_version: "encryptionkey@0"
I20260926 02:07:14.770499 22348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:14.794997 22348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:14.799597 22348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:14.799939 22348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:14.801086 22348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:14.802788 22348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:14.802904 22348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:14.803062 22348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:14.803135 22348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:14.837314 22348 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:39675
I20260926 02:07:14.837395 22475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:39675 every 8 connection(s)
I20260926 02:07:14.839071 22348 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:07:14.843148 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 22348
I20260926 02:07:14.843348 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:07:14.843876 19070 external_mini_cluster.cc:1428] Setting key da76e09956c8346f45c0d26d6de1599f
Sep 26 02:07:14 dist-test-slave-03rn krb5kdc[22108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388434, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:14.860810 22478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:14.841963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:43451 (local address 127.18.159.190:41369)
0926 02:07:14.842380 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:14.842385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:14.843734 (+ 1349us) server_negotiation.cc:407] Connection header received
0926 02:07:14.845734 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:14.845739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:14.845820 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:14.845951 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:14.847596 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:14.848401 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:14.850021 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:14.850425 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:14.850569 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:14.854738 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:14.854744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:14.854748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:14.854783 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:14.857519 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:14.858382 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:14.858388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:14.858392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:14.858527 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:14.858969 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:14.858975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:14.858978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:14.859276 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:14.859430 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:14.860198 (+ 768us) server_negotiation.cc:299] Negotiation successful
0926 02:07:14.860423 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":208,"threads_started":1}
WARNING: no policy specified for kudu/127.18.159.131@KRBTEST.COM; defaulting to no policy
I20260926 02:07:14.862051 22479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:14.842642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:41369 (local address 127.18.159.130:43451)
0926 02:07:14.843528 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:14.843586 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:14.845428 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:14.846217 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:14.846227 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:14.846784 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:14.847400 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:14.847418 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:14.848557 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:14.848562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:14.849858 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:14.849870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:14.850665 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:14.850671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:14.850913 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:14.851661 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:14.851696 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:14.854476 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:14.857685 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:14.857695 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:14.857714 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:14.858255 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:14.858639 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:14.858644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:14.858648 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:14.858836 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:14.859403 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:14.859412 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:14.859791 (+ 379us) client_negotiation.cc:769] Sending connection context
0926 02:07:14.860075 (+ 284us) client_negotiation.cc:240] Negotiation successful
0926 02:07:14.860700 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":361,"threads_started":1}
I20260926 02:07:14.863844 22476 heartbeater.cc:347] Connected to a master server at 127.18.159.190:41369
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.131@KRBTEST.COM" created.
I20260926 02:07:14.864303 22476 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:14.865154 22476 heartbeater.cc:511] Master 127.18.159.190:41369 requested a full tablet report, sending...
I20260926 02:07:14.866797 22153 ts_manager.cc:194] Registered new tserver with Master: 9b88486ad00c4e9d8d5583edfab8be26 (127.18.159.130:39675)
I20260926 02:07:14.867719 22153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:43451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
I20260926 02:07:14.887866 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:41369
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:15.231679 22482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:15.232012 22482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:15.232111 22482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:15.239980 22482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:15.240125 22482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:15.240362 22482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:07:15.251868 22482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:41369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:15.253443 22482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:15.254961 22482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:15.270569 22487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:15.270576 22490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:15.270569 22488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:15.271414 22482 server_base.cc:1034] running on GCE node
I20260926 02:07:15.272128 22482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:15.273285 22482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:15.274540 22482 hybrid_clock.cc:648] HybridClock initialized: now 1790388435274474 us; error 59 us; skew 500 ppm
I20260926 02:07:15.274927 22482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:15 dist-test-slave-03rn krb5kdc[22108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388435, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:15.278995 22482 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:07:15.280833 22482 webserver.cc:460] Webserver started at http://127.18.159.131:45223/ using document root <none> and password file <none>
I20260926 02:07:15.281633 22482 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:15.281750 22482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:15.282130 22482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:15.285526 22482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "19893797b80148ca9544be8fad533505"
format_stamp: "Formatted at 2026-09-26 02:07:15 on dist-test-slave-03rn"
server_key: "53896944b12de2f9769c0d5480446cc9"
server_key_iv: "5e5970c56d864402ee8ac4eca675847b"
server_key_version: "encryptionkey@0"
I20260926 02:07:15.286722 22482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "19893797b80148ca9544be8fad533505"
format_stamp: "Formatted at 2026-09-26 02:07:15 on dist-test-slave-03rn"
server_key: "53896944b12de2f9769c0d5480446cc9"
server_key_iv: "5e5970c56d864402ee8ac4eca675847b"
server_key_version: "encryptionkey@0"
I20260926 02:07:15.292999 22482 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 02:07:15.298007 22497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:15.299871 22482 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260926 02:07:15.300060 22482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "19893797b80148ca9544be8fad533505"
format_stamp: "Formatted at 2026-09-26 02:07:15 on dist-test-slave-03rn"
server_key: "53896944b12de2f9769c0d5480446cc9"
server_key_iv: "5e5970c56d864402ee8ac4eca675847b"
server_key_version: "encryptionkey@0"
I20260926 02:07:15.300321 22482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:15.325213 22482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:15.329887 22482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:15.330178 22482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:15.331292 22482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:15.332859 22342 heartbeater.cc:503] Master 127.18.159.190:41369 was elected leader, sending a full tablet report...
I20260926 02:07:15.332959 22482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:15.333042 22482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:15.333171 22482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:15.333230 22482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:15.372102 22482 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:34335
I20260926 02:07:15.372263 22609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:34335 every 8 connection(s)
I20260926 02:07:15.373553 22482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:07:15.379313 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 22482
I20260926 02:07:15.379669 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:07:15.380319 19070 external_mini_cluster.cc:1428] Setting key 79a3436e9b07c8d35cb6277eaa6e46e3
Sep 26 02:07:15 dist-test-slave-03rn krb5kdc[22108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388435, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:15.398875 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:15.376711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:56059 (local address 127.18.159.190:41369)
0926 02:07:15.377181 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:15.377185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:15.378104 (+ 919us) server_negotiation.cc:407] Connection header received
0926 02:07:15.381079 (+ 2975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:15.381086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:15.381190 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:15.381314 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:15.383900 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:15.384754 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:15.386437 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:15.386856 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:15.387011 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:15.391482 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:15.391489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:15.391494 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:15.391538 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:15.394760 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:15.396160 (+ 1400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:15.396193 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:15.396197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:15.396349 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:15.396855 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:15.396860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:15.396864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:15.397173 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:15.397342 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:15.398173 (+ 831us) server_negotiation.cc:299] Negotiation successful
0926 02:07:15.398437 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":244,"threads_started":1}
I20260926 02:07:15.399917 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:15.377227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:41369 (local address 127.18.159.131:56059)
0926 02:07:15.377937 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:15.377975 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:15.380538 (+ 2563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:15.381823 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:15.381838 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:15.382792 (+ 954us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:15.383719 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:15.383754 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:15.384912 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:15.384918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:15.386253 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:15.386265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:15.387116 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:15.387123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:15.387416 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:15.388416 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:15.388443 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:15.391167 (+ 2724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:15.395205 (+ 4038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:15.395216 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:15.395385 (+ 169us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:15.396015 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:15.396481 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:15.396486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:15.396490 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:15.396728 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:15.397329 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:15.397338 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:15.397733 (+ 395us) client_negotiation.cc:769] Sending connection context
0926 02:07:15.398043 (+ 310us) client_negotiation.cc:240] Negotiation successful
0926 02:07:15.398701 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":275,"threads_started":1}
I20260926 02:07:15.401966 22610 heartbeater.cc:347] Connected to a master server at 127.18.159.190:41369
I20260926 02:07:15.402510 22610 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:15.403920 22610 heartbeater.cc:511] Master 127.18.159.190:41369 requested a full tablet report, sending...
I20260926 02:07:15.405902 22153 ts_manager.cc:194] Registered new tserver with Master: 19893797b80148ca9544be8fad533505 (127.18.159.131:34335)
I20260926 02:07:15.406952 22153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:56059
I20260926 02:07:15.410945 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:07:15 dist-test-slave-03rn krb5kdc[22108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388435, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 02:07:15 dist-test-slave-03rn krb5kdc[22108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388435, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:15.466934 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:15.450796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.18.159.190:41369)
0926 02:07:15.450999 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:15.451003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:15.451562 (+ 559us) server_negotiation.cc:407] Connection header received
0926 02:07:15.451709 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:15.451713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:15.451801 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:15.451902 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:15.453603 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:15.454622 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:15.456080 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:15.456601 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:15.456720 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:15.460917 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:15.460923 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:15.460926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:15.460973 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:15.464461 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:15.465266 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:15.465271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:15.465273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:15.465396 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:15.465767 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:15.465770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:15.465772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:15.466087 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:15.466262 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:15.466533 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 02:07:15.466710 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 02:07:15.467559 22153 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53520
I20260926 02:07:15.469517 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 22214
Sep 26 02:07:15 dist-test-slave-03rn krb5kdc[22108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388435, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:07:15.488405 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:15.470486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52624 (local address 127.18.159.129:45257)
0926 02:07:15.471113 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:15.471122 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:15.471156 (+ 34us) server_negotiation.cc:407] Connection header received
0926 02:07:15.471230 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:15.471239 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:15.471492 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:15.471663 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:15.473615 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:15.474899 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:15.476337 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:15.476919 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:15.477026 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:15.481288 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:15.481303 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:15.481320 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:15.481365 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:15.484807 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:15.485668 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:15.485676 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:15.485678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:15.485830 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:15.486368 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:15.486372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:15.486374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:15.487065 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:15.487388 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:15.487745 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 02:07:15.487986 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":288,"threads_started":1}
W20260926 02:07:15.489816 22337 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52624
W20260926 02:07:15.490978 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 22214: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:07:15.491205 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 22214
I20260926 02:07:15.507457 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 22348
Sep 26 02:07:15 dist-test-slave-03rn krb5kdc[22108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388435, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.18.159.130@KRBTEST.COM
I20260926 02:07:15.525878 22623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:15.508426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49530 (local address 127.18.159.130:39675)
0926 02:07:15.508908 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:15.508914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:15.508940 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:07:15.509005 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:15.509012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:15.509228 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:15.509416 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:15.511552 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:15.513111 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:15.514469 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:15.515067 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:15.515220 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:15.519545 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:15.519559 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:15.519575 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:15.519623 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:15.522543 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:15.523256 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:15.523264 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:15.523266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:15.523401 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:15.523796 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:15.523802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:15.523805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:15.524362 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:15.524835 (+ 473us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:15.525193 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 02:07:15.525506 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":237,"threads_started":1}
W20260926 02:07:15.527382 22471 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49530
W20260926 02:07:15.528594 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 22348: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:07:15.528772 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 22348
I20260926 02:07:15.544727 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 22482
Sep 26 02:07:15 dist-test-slave-03rn krb5kdc[22108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388435, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.18.159.131@KRBTEST.COM
I20260926 02:07:15.562371 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:15.545709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54386 (local address 127.18.159.131:34335)
0926 02:07:15.546319 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:15.546327 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:15.546356 (+ 29us) server_negotiation.cc:407] Connection header received
0926 02:07:15.546424 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:15.546430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:15.546648 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:15.546785 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:15.548078 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:15.549213 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:15.550454 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:15.551062 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:15.551170 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:15.556071 (+ 4901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:15.556087 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:15.556103 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:15.556147 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:15.559297 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:15.560098 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:15.560106 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:15.560109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:15.560398 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:15.560829 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:15.560833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:15.560835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:15.561208 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:15.561457 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:15.561775 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 02:07:15.562072 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":338,"threads_started":1}
W20260926 02:07:15.563715 22605 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54386
W20260926 02:07:15.565196 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 22482: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:07:15.565392 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 22482
I20260926 02:07:15.581823 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 22121
I20260926 02:07:15.583169 22183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:15.692246 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 22121
2026-09-26T02:07:15Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3773 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:07:15 dist-test-slave-03rn krb5kdc[22628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:07:15 dist-test-slave-03rn krb5kdc[22628](info): set up 2 sockets
Sep 26 02:07:15 dist-test-slave-03rn krb5kdc[22628](info): commencing operation
krb5kdc: starting...
W20260926 02:07:16.789132 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:07:16 dist-test-slave-03rn krb5kdc[22628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:07:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:07:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:07:17.001516 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:35589
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:35589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:17.342538 22641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:17.342867 22641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:17.343032 22641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:17.350492 22641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:07:17.350615 22641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:17.350697 22641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:17.350809 22641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:07:17.350894 22641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:07:17.361184 22641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:35589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:35589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:17.362798 22641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:17.364800 22641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:17.378391 22649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:17.378350 22646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:17.378443 22647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:17.382215 22641 server_base.cc:1034] running on GCE node
I20260926 02:07:17.383193 22641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:17.384920 22641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:17.386171 22641 hybrid_clock.cc:648] HybridClock initialized: now 1790388437386117 us; error 44 us; skew 500 ppm
I20260926 02:07:17.386576 22641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:17 dist-test-slave-03rn krb5kdc[22628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388437, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:17.390228 22641 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:07:17.392027 22641 webserver.cc:460] Webserver started at http://127.18.159.190:39239/ using document root <none> and password file <none>
I20260926 02:07:17.392860 22641 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:17.393008 22641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:17.393357 22641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:17.396593 22641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "ffb570c91b9547c89ed77925a556de01"
format_stamp: "Formatted at 2026-09-26 02:07:17 on dist-test-slave-03rn"
server_key: "4c191672603e231ec055d6050e429619"
server_key_iv: "34e92c4bc3b6025719a9617505044d78"
server_key_version: "encryptionkey@0"
I20260926 02:07:17.397459 22641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "ffb570c91b9547c89ed77925a556de01"
format_stamp: "Formatted at 2026-09-26 02:07:17 on dist-test-slave-03rn"
server_key: "4c191672603e231ec055d6050e429619"
server_key_iv: "34e92c4bc3b6025719a9617505044d78"
server_key_version: "encryptionkey@0"
I20260926 02:07:17.403429 22641 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:07:17.407778 22656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:17.409628 22641 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:07:17.409885 22641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "ffb570c91b9547c89ed77925a556de01"
format_stamp: "Formatted at 2026-09-26 02:07:17 on dist-test-slave-03rn"
server_key: "4c191672603e231ec055d6050e429619"
server_key_iv: "34e92c4bc3b6025719a9617505044d78"
server_key_version: "encryptionkey@0"
I20260926 02:07:17.410132 22641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:17.452700 22641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:17.457273 22641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:17.457597 22641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:17.477646 22641 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:35589
I20260926 02:07:17.477679 22707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:35589 every 8 connection(s)
I20260926 02:07:17.479153 22641 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:07:17.483464 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 22641
I20260926 02:07:17.483608 22708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:17.483719 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:07:17.484541 19070 external_mini_cluster.cc:1428] Setting key 66333c584a140934ea7ffc2f2468bc33
I20260926 02:07:17.499042 22708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01: Bootstrap starting.
Sep 26 02:07:17 dist-test-slave-03rn krb5kdc[22628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:17.502987 22708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:17.504199 22708 log.cc:824] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:17.507266 22708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01: No bootstrap required, opened a new log
I20260926 02:07:17.510735 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:17.486784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.18.159.190:35589)
0926 02:07:17.487728 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:17.487740 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:17.487784 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:07:17.490324 (+ 2540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:17.490356 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:17.490846 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:17.491238 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:17.493324 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:17.494725 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:17.496726 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:17.497550 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:17.497707 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:17.502703 (+ 4996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:17.502714 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:17.502733 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:17.502783 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:17.506173 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:17.506994 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:17.507001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:17.507008 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:17.507180 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:17.507650 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:17.507654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:17.507656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:17.508310 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:17.508637 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:17.509139 (+ 502us) server_negotiation.cc:299] Negotiation successful
0926 02:07:17.509596 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":488,"threads_started":1}
I20260926 02:07:17.515681 22708 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffb570c91b9547c89ed77925a556de01" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35589 } }
I20260926 02:07:17.516120 22708 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:17.516283 22708 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffb570c91b9547c89ed77925a556de01, State: Initialized, Role: FOLLOWER
I20260926 02:07:17.516855 22708 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffb570c91b9547c89ed77925a556de01" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35589 } }
I20260926 02:07:17.517104 22708 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:07:17.517239 22708 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:07:17.517405 22708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:17.519594 22708 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffb570c91b9547c89ed77925a556de01" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35589 } }
I20260926 02:07:17.520205 22708 leader_election.cc:302] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffb570c91b9547c89ed77925a556de01; no voters:
I20260926 02:07:17.520874 22708 leader_election.cc:288] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:07:17.520988 22713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:17.521626 22713 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01 [term 1 LEADER]: Becoming Leader. State: Replica: ffb570c91b9547c89ed77925a556de01, State: Running, Role: LEADER
I20260926 02:07:17.522212 22713 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffb570c91b9547c89ed77925a556de01" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35589 } }
I20260926 02:07:17.523299 22708 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:07:17.525395 22715 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffb570c91b9547c89ed77925a556de01. Latest consensus state: current_term: 1 leader_uuid: "ffb570c91b9547c89ed77925a556de01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffb570c91b9547c89ed77925a556de01" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35589 } } }
I20260926 02:07:17.526032 22715 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01 [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:17.527940 22714 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffb570c91b9547c89ed77925a556de01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffb570c91b9547c89ed77925a556de01" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35589 } } }
I20260926 02:07:17.528206 22721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:07:17.528690 22714 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01 [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:17.533812 22721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:07:17.544966 22721 catalog_manager.cc:1347] Generated new cluster ID: 27774bf36c124673935c2f7521156f54
I20260926 02:07:17.545099 22721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:07:17.591624 22721 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:07:17.593086 22721 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:07:17.601783 22721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffb570c91b9547c89ed77925a556de01: Generated new TSK 0
I20260926 02:07:17.602854 22721 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.159.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
I20260926 02:07:17.656541 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35589
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:18.001118 22734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:18.001487 22734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:18.001658 22734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:18.009243 22734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:18.009428 22734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:18.009640 22734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:07:18.020208 22734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:18.021600 22734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:18.023020 22734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:18.036969 22739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:18.036943 22740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:18.037022 22742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:18.037674 22734 server_base.cc:1034] running on GCE node
I20260926 02:07:18.038337 22734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:18.039391 22734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:18.040647 22734 hybrid_clock.cc:648] HybridClock initialized: now 1790388438040582 us; error 57 us; skew 500 ppm
I20260926 02:07:18.041015 22734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:18 dist-test-slave-03rn krb5kdc[22628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388438, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:18.044478 22734 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:07:18.046149 22734 webserver.cc:460] Webserver started at http://127.18.159.129:44095/ using document root <none> and password file <none>
I20260926 02:07:18.046939 22734 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:18.047092 22734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:18.047432 22734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:18.050566 22734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "d900c336e3754f698de4c4a975e7362c"
format_stamp: "Formatted at 2026-09-26 02:07:18 on dist-test-slave-03rn"
server_key: "ec719ae491e7cb10d589d6734fad26ae"
server_key_iv: "1e7e05b89f1b18a17a2fec2a055a6ee8"
server_key_version: "encryptionkey@0"
I20260926 02:07:18.051431 22734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "d900c336e3754f698de4c4a975e7362c"
format_stamp: "Formatted at 2026-09-26 02:07:18 on dist-test-slave-03rn"
server_key: "ec719ae491e7cb10d589d6734fad26ae"
server_key_iv: "1e7e05b89f1b18a17a2fec2a055a6ee8"
server_key_version: "encryptionkey@0"
I20260926 02:07:18.057183 22734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:07:18.061062 22749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:18.062620 22734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:07:18.062835 22734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "d900c336e3754f698de4c4a975e7362c"
format_stamp: "Formatted at 2026-09-26 02:07:18 on dist-test-slave-03rn"
server_key: "ec719ae491e7cb10d589d6734fad26ae"
server_key_iv: "1e7e05b89f1b18a17a2fec2a055a6ee8"
server_key_version: "encryptionkey@0"
I20260926 02:07:18.063107 22734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:18.085413 22734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:18.090679 22734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:18.091029 22734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:18.092110 22734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:18.093604 22734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:18.093717 22734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:18.093881 22734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:18.093968 22734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:18.128674 22734 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:38653
I20260926 02:07:18.128706 22861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:38653 every 8 connection(s)
I20260926 02:07:18.130002 22734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:07:18.139796 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 22734
I20260926 02:07:18.140039 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:07:18.140823 19070 external_mini_cluster.cc:1428] Setting key c65bb0cebbcde13affa3fc5965870c84
Sep 26 02:07:18 dist-test-slave-03rn krb5kdc[22628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388438, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:18.152326 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:18.132621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:59415 (local address 127.18.159.190:35589)
0926 02:07:18.133059 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:18.133064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:18.134483 (+ 1419us) server_negotiation.cc:407] Connection header received
0926 02:07:18.136482 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:18.136487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:18.136559 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:18.136665 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:18.138263 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:18.139025 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:18.141560 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:18.142161 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:18.142303 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:18.146526 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:18.146532 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:18.146535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:18.146568 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:18.149174 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:18.149951 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:18.149959 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:18.149967 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:18.150080 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:18.150527 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:18.150533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:18.150536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:18.150796 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:18.150967 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:18.151693 (+ 726us) server_negotiation.cc:299] Negotiation successful
0926 02:07:18.151924 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":333,"thread_start_us":229,"threads_started":1}
I20260926 02:07:18.153368 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:18.133313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35589 (local address 127.18.159.129:59415)
0926 02:07:18.134251 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:18.134306 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:18.136113 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:18.136887 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:18.136896 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:18.137418 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:18.138051 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:18.138072 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:18.139180 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:18.139184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:18.141355 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:18.141369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:18.142385 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:18.142391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:18.142619 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:18.143413 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:18.143449 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:18.146266 (+ 2817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:18.149339 (+ 3073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:18.149350 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:18.149379 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:18.149811 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:18.150192 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:18.150197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:18.150201 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:18.150408 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:18.150945 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:18.150953 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:18.151320 (+ 367us) client_negotiation.cc:769] Sending connection context
0926 02:07:18.151604 (+ 284us) client_negotiation.cc:240] Negotiation successful
0926 02:07:18.152162 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":367,"threads_started":1}
I20260926 02:07:18.155222 22862 heartbeater.cc:347] Connected to a master server at 127.18.159.190:35589
I20260926 02:07:18.155577 22862 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:18.156452 22862 heartbeater.cc:511] Master 127.18.159.190:35589 requested a full tablet report, sending...
I20260926 02:07:18.158790 22673 ts_manager.cc:194] Registered new tserver with Master: d900c336e3754f698de4c4a975e7362c (127.18.159.129:38653)
WARNING: no policy specified for kudu/127.18.159.130@KRBTEST.COM; defaulting to no policy
I20260926 02:07:18.160422 22673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:59415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
I20260926 02:07:18.187811 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35589
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:18.550578 22868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:18.550887 22868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:18.550980 22868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:18.558357 22868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:18.558496 22868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:18.558611 22868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:07:18.568967 22868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:18.570389 22868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:18.571771 22868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:18.585328 22873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:18.585346 22876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:18.585460 22874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:18.586959 22868 server_base.cc:1034] running on GCE node
I20260926 02:07:18.587833 22868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:18.589035 22868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:18.590257 22868 hybrid_clock.cc:648] HybridClock initialized: now 1790388438590213 us; error 43 us; skew 500 ppm
I20260926 02:07:18.590616 22868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:18 dist-test-slave-03rn krb5kdc[22628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388438, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:18.594384 22868 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:07:18.595970 22868 webserver.cc:460] Webserver started at http://127.18.159.130:37373/ using document root <none> and password file <none>
I20260926 02:07:18.596738 22868 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:18.596866 22868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:18.597196 22868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:18.600392 22868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "54ed455d9f844d3593b15451b5e9a077"
format_stamp: "Formatted at 2026-09-26 02:07:18 on dist-test-slave-03rn"
server_key: "afc61a9743d8662caff734854d1d211e"
server_key_iv: "fd64dd70744a37fbef64fd7a4a018235"
server_key_version: "encryptionkey@0"
I20260926 02:07:18.601223 22868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "54ed455d9f844d3593b15451b5e9a077"
format_stamp: "Formatted at 2026-09-26 02:07:18 on dist-test-slave-03rn"
server_key: "afc61a9743d8662caff734854d1d211e"
server_key_iv: "fd64dd70744a37fbef64fd7a4a018235"
server_key_version: "encryptionkey@0"
I20260926 02:07:18.607267 22868 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 02:07:18.611361 22883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:18.613359 22868 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:07:18.613590 22868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "54ed455d9f844d3593b15451b5e9a077"
format_stamp: "Formatted at 2026-09-26 02:07:18 on dist-test-slave-03rn"
server_key: "afc61a9743d8662caff734854d1d211e"
server_key_iv: "fd64dd70744a37fbef64fd7a4a018235"
server_key_version: "encryptionkey@0"
I20260926 02:07:18.613821 22868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:18.631207 22868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:18.635968 22868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:18.636332 22868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:18.637359 22868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:18.638736 22868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:18.638836 22868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:18.639005 22868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:18.639079 22868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:18.673691 22868 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:38101
I20260926 02:07:18.673734 22995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:38101 every 8 connection(s)
I20260926 02:07:18.675092 22868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:07:18.681380 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 22868
I20260926 02:07:18.681600 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:07:18.682030 19070 external_mini_cluster.cc:1428] Setting key 85ec30bd69f24c0685dd1eaf67370b34
Sep 26 02:07:18 dist-test-slave-03rn krb5kdc[22628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388438, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:18.699203 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:18.678067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:59471 (local address 127.18.159.190:35589)
0926 02:07:18.678552 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:18.678558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:18.679744 (+ 1186us) server_negotiation.cc:407] Connection header received
0926 02:07:18.681836 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:18.681842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:18.681944 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:18.682072 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:18.684300 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:18.685503 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:18.687291 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:18.688043 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:18.688237 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:18.692559 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:18.692564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:18.692567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:18.692604 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:18.695589 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:18.696599 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:18.696604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:18.696606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:18.696747 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:18.697356 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:18.697361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:18.697363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:18.697644 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:18.697805 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:18.698560 (+ 755us) server_negotiation.cc:299] Negotiation successful
0926 02:07:18.698805 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":263,"threads_started":1}
I20260926 02:07:18.700392 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:18.678750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35589 (local address 127.18.159.130:59471)
0926 02:07:18.679548 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:18.679595 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:18.681460 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:18.682350 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:18.682360 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:18.683078 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:18.683984 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:18.684032 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:18.685714 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:18.685718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:18.687097 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:18.687113 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:18.688388 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:18.688394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:18.688634 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:18.689411 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:18.689436 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:18.692269 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:18.695786 (+ 3517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:18.695798 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:18.695819 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:18.696434 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:18.696880 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:18.696886 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:18.696889 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:18.697200 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:18.697783 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:18.697792 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:18.698192 (+ 400us) client_negotiation.cc:769] Sending connection context
0926 02:07:18.698483 (+ 291us) client_negotiation.cc:240] Negotiation successful
0926 02:07:18.699074 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":307,"threads_started":1}
WARNING: no policy specified for kudu/127.18.159.131@KRBTEST.COM; defaulting to no policy
I20260926 02:07:18.702286 22996 heartbeater.cc:347] Connected to a master server at 127.18.159.190:35589
I20260926 02:07:18.702659 22996 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:18.703593 22996 heartbeater.cc:511] Master 127.18.159.190:35589 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.131@KRBTEST.COM" created.
I20260926 02:07:18.705590 22673 ts_manager.cc:194] Registered new tserver with Master: 54ed455d9f844d3593b15451b5e9a077 (127.18.159.130:38101)
I20260926 02:07:18.706945 22673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:59471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
I20260926 02:07:18.731904 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35589
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:19.079823 23002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:19.080236 23002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:19.080430 23002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:19.088420 23002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:19.088567 23002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:19.088682 23002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:07:19.098721 23002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:19.100040 23002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:19.101349 23002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:19.115103 23007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:19.117700 23008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:19.117730 23010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:19.117843 23002 server_base.cc:1034] running on GCE node
I20260926 02:07:19.118628 23002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:19.119647 23002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:19.120893 23002 hybrid_clock.cc:648] HybridClock initialized: now 1790388439120804 us; error 58 us; skew 500 ppm
I20260926 02:07:19.121294 23002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:19 dist-test-slave-03rn krb5kdc[22628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388439, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:19.125229 23002 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:07:19.127079 23002 webserver.cc:460] Webserver started at http://127.18.159.131:42235/ using document root <none> and password file <none>
I20260926 02:07:19.127862 23002 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:19.128026 23002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:19.128413 23002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:19.131426 23002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "f9473e2ae2ef448db69602c04f56e4aa"
format_stamp: "Formatted at 2026-09-26 02:07:19 on dist-test-slave-03rn"
server_key: "d8c52725557ee672a81626d7e8ab4d06"
server_key_iv: "f27cae5cd01a6750d9f9cdf0559001df"
server_key_version: "encryptionkey@0"
I20260926 02:07:19.132424 23002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "f9473e2ae2ef448db69602c04f56e4aa"
format_stamp: "Formatted at 2026-09-26 02:07:19 on dist-test-slave-03rn"
server_key: "d8c52725557ee672a81626d7e8ab4d06"
server_key_iv: "f27cae5cd01a6750d9f9cdf0559001df"
server_key_version: "encryptionkey@0"
I20260926 02:07:19.138374 23002 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:07:19.142330 23017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:19.143994 23002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:07:19.144259 23002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "f9473e2ae2ef448db69602c04f56e4aa"
format_stamp: "Formatted at 2026-09-26 02:07:19 on dist-test-slave-03rn"
server_key: "d8c52725557ee672a81626d7e8ab4d06"
server_key_iv: "f27cae5cd01a6750d9f9cdf0559001df"
server_key_version: "encryptionkey@0"
I20260926 02:07:19.144519 23002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:19.159742 23002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:19.163882 22862 heartbeater.cc:503] Master 127.18.159.190:35589 was elected leader, sending a full tablet report...
I20260926 02:07:19.165477 23002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:19.165840 23002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:19.166999 23002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:19.168736 23002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:19.168819 23002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:19.168915 23002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:19.168965 23002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:19.207056 23002 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:43267
I20260926 02:07:19.207118 23129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:43267 every 8 connection(s)
I20260926 02:07:19.208608 23002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:07:19.216373 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 23002
I20260926 02:07:19.216570 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:07:19.217085 19070 external_mini_cluster.cc:1428] Setting key f2ef0d0f7f54cc58823c0cfdc281672c
Sep 26 02:07:19 dist-test-slave-03rn krb5kdc[22628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388439, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:19.235344 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:19.211503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:45135 (local address 127.18.159.190:35589)
0926 02:07:19.211986 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:19.211993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:19.213208 (+ 1215us) server_negotiation.cc:407] Connection header received
0926 02:07:19.215205 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:19.215210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:19.215279 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:19.215375 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:19.217765 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:19.218637 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:19.220616 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:19.221018 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:19.221166 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:19.228865 (+ 7699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:19.228871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:19.228874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:19.228921 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:19.231742 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:19.232770 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:19.232778 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:19.232785 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:19.232944 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:19.233470 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:19.233475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:19.233478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:19.233757 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:19.233917 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:19.234714 (+ 797us) server_negotiation.cc:299] Negotiation successful
0926 02:07:19.234950 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":215,"threads_started":1}
I20260926 02:07:19.236523 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:19.212199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35589 (local address 127.18.159.131:45135)
0926 02:07:19.213042 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:19.213088 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:19.214857 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:19.215636 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:19.215645 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:19.216576 (+ 931us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:19.217538 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:19.217569 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:19.218835 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:19.218852 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:19.220463 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:19.220478 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:19.221264 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:19.221274 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:19.221537 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:19.226076 (+ 4539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:19.226105 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:19.228563 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:19.231935 (+ 3372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:19.231948 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:19.231974 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:19.232611 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:19.233111 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:19.233116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:19.233120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:19.233340 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:19.233904 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:19.233913 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:19.234311 (+ 398us) client_negotiation.cc:769] Sending connection context
0926 02:07:19.234611 (+ 300us) client_negotiation.cc:240] Negotiation successful
0926 02:07:19.235189 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":325,"threads_started":1}
I20260926 02:07:19.238137 23130 heartbeater.cc:347] Connected to a master server at 127.18.159.190:35589
I20260926 02:07:19.238511 23130 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:19.239315 23130 heartbeater.cc:511] Master 127.18.159.190:35589 requested a full tablet report, sending...
I20260926 02:07:19.240940 22673 ts_manager.cc:194] Registered new tserver with Master: f9473e2ae2ef448db69602c04f56e4aa (127.18.159.131:43267)
I20260926 02:07:19.241755 22673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:45135
I20260926 02:07:19.246119 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:07:19 dist-test-slave-03rn krb5kdc[22628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:07:19 dist-test-slave-03rn krb5kdc[22628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:07:19.299271 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:19.282355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58424 (local address 127.18.159.129:38653)
0926 02:07:19.282970 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:19.282978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:19.283006 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:07:19.283105 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:19.283111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:19.283344 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:19.283491 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:19.285199 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:19.286309 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:19.287723 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:19.288501 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:19.288618 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:19.293119 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:19.293137 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:19.293163 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:19.293209 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:19.296297 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:19.297092 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:19.297101 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:19.297103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:19.297240 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:19.297665 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:19.297669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:19.297671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:19.298062 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:19.298295 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:19.298616 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 02:07:19.298816 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":315,"threads_started":1}
Sep 26 02:07:19 dist-test-slave-03rn krb5kdc[22628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:19.318555 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:19.303892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40478 (local address 127.18.159.190:35589)
0926 02:07:19.304143 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:19.304150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:19.304455 (+ 305us) server_negotiation.cc:407] Connection header received
0926 02:07:19.304667 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:19.304672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:19.304748 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:19.304873 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:19.306181 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:19.306922 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:19.308245 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:19.308667 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:19.308803 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:19.312739 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:19.312745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:19.312748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:19.312787 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:19.315535 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:19.316435 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:19.316440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:19.316442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:19.316573 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:19.316977 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:19.316981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:19.316983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:19.317355 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:19.317585 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:19.317970 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 02:07:19.318241 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260926 02:07:19.319195 22673 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40478
I20260926 02:07:19.320469 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 22734
I20260926 02:07:19.334591 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:19.321616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58432 (local address 127.18.159.129:38653)
0926 02:07:19.321852 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:19.321858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:19.322038 (+ 180us) server_negotiation.cc:407] Connection header received
0926 02:07:19.322257 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:19.322261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:19.322355 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:19.322465 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:19.323840 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:19.324688 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:19.326090 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:19.326533 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:19.326679 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:19.329327 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:19.329332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:19.329335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:19.329371 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:19.331979 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:19.332768 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:19.332778 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:19.332780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:19.332913 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:19.333289 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:19.333294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:19.333297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:19.333599 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:19.333829 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:19.334096 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 02:07:19.334307 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 02:07:19.335232 22857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:19.420955 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 22734
I20260926 02:07:19.438015 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 22868
Sep 26 02:07:19 dist-test-slave-03rn krb5kdc[22628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.130@KRBTEST.COM
I20260926 02:07:19.455681 23143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:19.438945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.18.159.130:38101)
0926 02:07:19.439570 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:19.439578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:19.439607 (+ 29us) server_negotiation.cc:407] Connection header received
0926 02:07:19.439669 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:19.439675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:19.439898 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:19.440057 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:19.441437 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:19.442503 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:19.443922 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:19.444750 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:19.444879 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:19.449658 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:19.449673 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:19.449691 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:19.449733 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:19.452797 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:19.453560 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:19.453569 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:19.453571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:19.453703 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:19.454095 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:19.454099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:19.454101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:19.454452 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:19.454702 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:19.455144 (+ 442us) server_negotiation.cc:299] Negotiation successful
0926 02:07:19.455371 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":324,"threads_started":1}
I20260926 02:07:19.457029 22991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:19.546703 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 22868
I20260926 02:07:19.562825 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 23002
Sep 26 02:07:19 dist-test-slave-03rn krb5kdc[22628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.131@KRBTEST.COM
I20260926 02:07:19.579895 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:19.563892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47060 (local address 127.18.159.131:43267)
0926 02:07:19.564557 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:19.564564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:19.564596 (+ 32us) server_negotiation.cc:407] Connection header received
0926 02:07:19.564654 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:19.564661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:19.564882 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:19.565013 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:19.566323 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:19.567400 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:19.568919 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:19.569518 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:19.569633 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:19.574291 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:19.574306 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:19.574323 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:19.574376 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:19.577224 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:19.577919 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:19.577927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:19.577929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:19.578053 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:19.578432 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:19.578436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:19.578439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:19.578832 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:19.579117 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:19.579390 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 02:07:19.579599 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":372,"threads_started":1}
I20260926 02:07:19.581369 23125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:19.664701 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 23002
I20260926 02:07:19.680616 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 22641
I20260926 02:07:19.681823 22703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:19.784593 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 22641
2026-09-26T02:07:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4093 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:07:19 dist-test-slave-03rn krb5kdc[23150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:07:19 dist-test-slave-03rn krb5kdc[23150](info): set up 2 sockets
Sep 26 02:07:19 dist-test-slave-03rn krb5kdc[23150](info): commencing operation
krb5kdc: starting...
W20260926 02:07:20.889976 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:07:21 dist-test-slave-03rn krb5kdc[23150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:07:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:07:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:07:21.125268 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:39577
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:39577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:21.497845 23163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:21.498185 23163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:21.498278 23163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:21.506834 23163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:07:21.507036 23163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:21.507085 23163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:21.507136 23163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:07:21.507190 23163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:07:21.518715 23163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:39577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:39577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:21.520059 23163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:21.521467 23163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:21.536325 23171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:21.536285 23168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:21.536252 23169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:21.537774 23163 server_base.cc:1034] running on GCE node
I20260926 02:07:21.538792 23163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:21.540643 23163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:21.541903 23163 hybrid_clock.cc:648] HybridClock initialized: now 1790388441541853 us; error 42 us; skew 500 ppm
I20260926 02:07:21.542330 23163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:21 dist-test-slave-03rn krb5kdc[23150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388441, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:21.545817 23163 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:07:21.547191 23163 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:21.547300 23163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:21.547675 23163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:21.551136 23163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "554a848bab004430a49adc03da9d3877"
format_stamp: "Formatted at 2026-09-26 02:07:21 on dist-test-slave-03rn"
server_key: "8676775926e0a624823162dd10bbdce5"
server_key_iv: "0aaea2a525534bd15309ca0ed79fcec0"
server_key_version: "encryptionkey@0"
I20260926 02:07:21.552742 23163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "554a848bab004430a49adc03da9d3877"
format_stamp: "Formatted at 2026-09-26 02:07:21 on dist-test-slave-03rn"
server_key: "8676775926e0a624823162dd10bbdce5"
server_key_iv: "0aaea2a525534bd15309ca0ed79fcec0"
server_key_version: "encryptionkey@0"
I20260926 02:07:21.558825 23163 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:07:21.563263 23177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:21.565256 23163 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:07:21.565500 23163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "554a848bab004430a49adc03da9d3877"
format_stamp: "Formatted at 2026-09-26 02:07:21 on dist-test-slave-03rn"
server_key: "8676775926e0a624823162dd10bbdce5"
server_key_iv: "0aaea2a525534bd15309ca0ed79fcec0"
server_key_version: "encryptionkey@0"
I20260926 02:07:21.565753 23163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:21.596146 23163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:21.601468 23163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:21.601791 23163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:21.623394 23163 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:39577
I20260926 02:07:21.623420 23228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:39577 every 8 connection(s)
I20260926 02:07:21.624446 23163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:07:21.629043 23229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:21.630466 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 23163
I20260926 02:07:21.630743 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:07:21.631731 19070 external_mini_cluster.cc:1428] Setting key ac5c5d730cca8c0ea81b48f73a91f6cf
I20260926 02:07:21.645509 23229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877: Bootstrap starting.
I20260926 02:07:21.649130 23229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877: Neither blocks nor log segments found. Creating new log.
Sep 26 02:07:21 dist-test-slave-03rn krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:21.650607 23229 log.cc:824] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:21.654349 23229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877: No bootstrap required, opened a new log
I20260926 02:07:21.662945 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:21.634288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44424 (local address 127.18.159.190:39577)
0926 02:07:21.635458 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:21.635479 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:21.635528 (+ 49us) server_negotiation.cc:407] Connection header received
0926 02:07:21.637324 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:21.637354 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:21.637879 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:21.638256 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:21.640354 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:21.641788 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:21.644584 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:21.645692 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:21.645967 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:21.652099 (+ 6132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:21.652109 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:21.652128 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:21.652239 (+ 111us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:21.655905 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:21.656935 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:21.656944 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:21.656952 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:21.657181 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:21.657703 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:21.657708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:21.657710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:21.658419 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:21.658814 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:21.659384 (+ 570us) server_negotiation.cc:299] Negotiation successful
0926 02:07:21.659938 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":905,"thread_start_us":482,"threads_started":1}
I20260926 02:07:21.664611 23229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554a848bab004430a49adc03da9d3877" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39577 } }
I20260926 02:07:21.665071 23229 raft_consensus.cc:374] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:21.665262 23229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554a848bab004430a49adc03da9d3877, State: Initialized, Role: FOLLOWER
I20260926 02:07:21.665957 23229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554a848bab004430a49adc03da9d3877" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39577 } }
I20260926 02:07:21.666210 23229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:07:21.666365 23229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:07:21.666640 23229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:21.669693 23229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554a848bab004430a49adc03da9d3877" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39577 } }
I20260926 02:07:21.670377 23229 leader_election.cc:302] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 554a848bab004430a49adc03da9d3877; no voters:
I20260926 02:07:21.671206 23229 leader_election.cc:288] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:07:21.671331 23234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:21.672592 23234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877 [term 1 LEADER]: Becoming Leader. State: Replica: 554a848bab004430a49adc03da9d3877, State: Running, Role: LEADER
I20260926 02:07:21.673218 23234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554a848bab004430a49adc03da9d3877" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39577 } }
I20260926 02:07:21.674052 23229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:07:21.677314 23235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "554a848bab004430a49adc03da9d3877" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554a848bab004430a49adc03da9d3877" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39577 } } }
I20260926 02:07:21.677289 23236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 554a848bab004430a49adc03da9d3877. Latest consensus state: current_term: 1 leader_uuid: "554a848bab004430a49adc03da9d3877" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554a848bab004430a49adc03da9d3877" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39577 } } }
I20260926 02:07:21.677788 23235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877 [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:21.677834 23236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877 [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:21.689960 23241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:07:21.695477 23241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:07:21.704717 23241 catalog_manager.cc:1347] Generated new cluster ID: d096fef2b48649bba39c7c990ebb0ced
I20260926 02:07:21.704877 23241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:07:21.726055 23241 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:07:21.727761 23241 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:07:21.738926 23241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877: Generated new TSK 0
I20260926 02:07:21.740257 23241 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.159.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
I20260926 02:07:21.791935 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:39577
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:22.140452 23255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:22.140765 23255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:22.140925 23255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:22.148535 23255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:22.148658 23255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:22.148826 23255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:07:22.159017 23255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:39577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:22.160475 23255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:22.161800 23255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:22.175787 23260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:22.175669 23261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:22.175859 23263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:22.176844 23255 server_base.cc:1034] running on GCE node
I20260926 02:07:22.177551 23255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:22.178664 23255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:22.179912 23255 hybrid_clock.cc:648] HybridClock initialized: now 1790388442179861 us; error 40 us; skew 500 ppm
I20260926 02:07:22.180363 23255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:22 dist-test-slave-03rn krb5kdc[23150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388442, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:22.183713 23255 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:07:22.185077 23255 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:22.185225 23255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:22.185554 23255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:22.189257 23255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "5c850c9f00ce450b9734e660210e5e6f"
format_stamp: "Formatted at 2026-09-26 02:07:22 on dist-test-slave-03rn"
server_key: "bffa415f938b94eeab9f9f4ff5d6338d"
server_key_iv: "353253aaeaad785ba00bfc02326d84af"
server_key_version: "encryptionkey@0"
I20260926 02:07:22.190122 23255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "5c850c9f00ce450b9734e660210e5e6f"
format_stamp: "Formatted at 2026-09-26 02:07:22 on dist-test-slave-03rn"
server_key: "bffa415f938b94eeab9f9f4ff5d6338d"
server_key_iv: "353253aaeaad785ba00bfc02326d84af"
server_key_version: "encryptionkey@0"
I20260926 02:07:22.195887 23255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 02:07:22.199756 23269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:22.201303 23255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:07:22.201516 23255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "5c850c9f00ce450b9734e660210e5e6f"
format_stamp: "Formatted at 2026-09-26 02:07:22 on dist-test-slave-03rn"
server_key: "bffa415f938b94eeab9f9f4ff5d6338d"
server_key_iv: "353253aaeaad785ba00bfc02326d84af"
server_key_version: "encryptionkey@0"
I20260926 02:07:22.201763 23255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:22.225386 23255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:22.230017 23255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:22.230334 23255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:22.231413 23255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:22.232960 23255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:22.233105 23255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:22.233251 23255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:22.233351 23255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:22.270401 23255 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:33849
I20260926 02:07:22.270457 23381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:33849 every 8 connection(s)
I20260926 02:07:22.271337 23255 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:07:22.275317 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 23255
I20260926 02:07:22.275568 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:07:22.276341 19070 external_mini_cluster.cc:1428] Setting key 95d06b75b9a1bec481b5b565dffc19a7
Sep 26 02:07:22 dist-test-slave-03rn krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388442, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:22.296811 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:22.274049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:47669 (local address 127.18.159.190:39577)
0926 02:07:22.274530 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:22.274534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:22.275796 (+ 1262us) server_negotiation.cc:407] Connection header received
0926 02:07:22.277867 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:22.277872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:22.277952 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:22.278079 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:22.279873 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:22.281087 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:22.282990 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:22.283752 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:22.283928 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:22.289392 (+ 5464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:22.289400 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:22.289403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:22.289444 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:22.292466 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:22.293765 (+ 1299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:22.293771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:22.293773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:22.293949 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:22.294608 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:22.294613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:22.294615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:22.294990 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:22.295116 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:22.296339 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0926 02:07:22.296542 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":247,"threads_started":1}
WARNING: no policy specified for kudu/127.18.159.130@KRBTEST.COM; defaulting to no policy
I20260926 02:07:22.298550 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:22.274641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:39577 (local address 127.18.159.129:47669)
0926 02:07:22.275583 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:22.275638 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:22.277572 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:22.278405 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:22.278414 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:22.279014 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:22.279657 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:22.279677 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:22.281311 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:22.281316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:22.282775 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:22.282793 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:22.284180 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:22.284191 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:22.284637 (+ 446us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:22.285552 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:22.285591 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:22.289068 (+ 3477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:22.292684 (+ 3616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:22.292698 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:22.292723 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:22.293569 (+ 846us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:22.294115 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:22.294123 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:22.294129 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:22.294440 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:22.295151 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:22.295163 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:22.295731 (+ 568us) client_negotiation.cc:769] Sending connection context
0926 02:07:22.296114 (+ 383us) client_negotiation.cc:240] Negotiation successful
0926 02:07:22.296826 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":364,"threads_started":1}
I20260926 02:07:22.300609 23382 heartbeater.cc:347] Connected to a master server at 127.18.159.190:39577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.130@KRBTEST.COM" created.
I20260926 02:07:22.301043 23382 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:22.302037 23382 heartbeater.cc:511] Master 127.18.159.190:39577 requested a full tablet report, sending...
I20260926 02:07:22.304934 23194 ts_manager.cc:194] Registered new tserver with Master: 5c850c9f00ce450b9734e660210e5e6f (127.18.159.129:33849)
I20260926 02:07:22.307243 23194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:47669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
I20260926 02:07:22.328881 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:39577
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:22.678068 23388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:22.678428 23388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:22.678615 23388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:22.686281 23388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:22.686475 23388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:22.686662 23388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:07:22.697361 23388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:39577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:22.698782 23388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:22.700285 23388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:22.713948 23393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:22.714555 23396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:22.714535 23394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:22.714982 23388 server_base.cc:1034] running on GCE node
I20260926 02:07:22.715654 23388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:22.716768 23388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:22.718039 23388 hybrid_clock.cc:648] HybridClock initialized: now 1790388442717974 us; error 35 us; skew 500 ppm
I20260926 02:07:22.718375 23388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:22 dist-test-slave-03rn krb5kdc[23150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388442, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:22.721846 23388 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:07:22.723204 23388 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:22.723353 23388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:22.723692 23388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:22.727268 23388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "975f7013538640789b0b2c73895ea547"
format_stamp: "Formatted at 2026-09-26 02:07:22 on dist-test-slave-03rn"
server_key: "7a1ffb20a2bab67c010971d900104310"
server_key_iv: "6dffb019fc970d709be23d4d8c5d606e"
server_key_version: "encryptionkey@0"
I20260926 02:07:22.728308 23388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "975f7013538640789b0b2c73895ea547"
format_stamp: "Formatted at 2026-09-26 02:07:22 on dist-test-slave-03rn"
server_key: "7a1ffb20a2bab67c010971d900104310"
server_key_iv: "6dffb019fc970d709be23d4d8c5d606e"
server_key_version: "encryptionkey@0"
I20260926 02:07:22.734205 23388 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:07:22.738179 23402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:22.739781 23388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:07:22.740039 23388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "975f7013538640789b0b2c73895ea547"
format_stamp: "Formatted at 2026-09-26 02:07:22 on dist-test-slave-03rn"
server_key: "7a1ffb20a2bab67c010971d900104310"
server_key_iv: "6dffb019fc970d709be23d4d8c5d606e"
server_key_version: "encryptionkey@0"
I20260926 02:07:22.740358 23388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:22.754630 23388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:22.759344 23388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:22.759660 23388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:22.760708 23388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:22.762146 23388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:22.762277 23388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:22.762427 23388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:22.762519 23388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:22.798578 23388 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:46251
I20260926 02:07:22.798629 23514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:46251 every 8 connection(s)
I20260926 02:07:22.799643 23388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:07:22.800727 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 23388
I20260926 02:07:22.800946 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:07:22.801374 19070 external_mini_cluster.cc:1428] Setting key 5035d10a88909c562b235bf32a3a693a
Sep 26 02:07:22 dist-test-slave-03rn krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388442, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.159.131@KRBTEST.COM; defaulting to no policy
I20260926 02:07:22.824934 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:22.802467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:41943 (local address 127.18.159.190:39577)
0926 02:07:22.803004 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:22.803010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:22.804443 (+ 1433us) server_negotiation.cc:407] Connection header received
0926 02:07:22.808259 (+ 3816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:22.808265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:22.808350 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:22.808464 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:22.811244 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:22.812558 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:22.814425 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:22.814872 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:22.815030 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:22.818815 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:22.818821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:22.818838 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:22.818873 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:22.821555 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:22.822387 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:22.822394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:22.822398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:22.822533 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:22.823003 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:22.823010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:22.823013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:22.823299 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:22.823462 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:22.824310 (+ 848us) server_negotiation.cc:299] Negotiation successful
0926 02:07:22.824548 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":256,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.131@KRBTEST.COM" created.
I20260926 02:07:22.826113 23518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:22.803152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:39577 (local address 127.18.159.130:41943)
0926 02:07:22.804213 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:22.804298 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:22.807700 (+ 3402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:22.808855 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:22.808873 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:22.809886 (+ 1013us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:22.810906 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:22.810954 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:22.812751 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:22.812756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:22.814188 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:22.814203 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:22.815160 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:22.815167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:22.815414 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:22.816213 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:22.816272 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:22.818526 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:22.821745 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:22.821756 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:22.821776 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:22.822239 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:22.822666 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:22.822672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:22.822675 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:22.822880 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:22.823452 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:22.823461 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:22.823922 (+ 461us) client_negotiation.cc:769] Sending connection context
0926 02:07:22.824235 (+ 313us) client_negotiation.cc:240] Negotiation successful
0926 02:07:22.824793 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":398,"threads_started":1}
I20260926 02:07:22.827807 23515 heartbeater.cc:347] Connected to a master server at 127.18.159.190:39577
I20260926 02:07:22.828120 23515 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:22.828913 23515 heartbeater.cc:511] Master 127.18.159.190:39577 requested a full tablet report, sending...
I20260926 02:07:22.830644 23194 ts_manager.cc:194] Registered new tserver with Master: 975f7013538640789b0b2c73895ea547 (127.18.159.130:46251)
I20260926 02:07:22.831672 23194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:41943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
I20260926 02:07:22.851325 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:39577
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:23.201594 23521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:23.201901 23521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:23.202001 23521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:23.210140 23521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:23.210275 23521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:23.210403 23521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:07:23.221498 23521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:39577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:23.222826 23521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:23.224254 23521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:23.238576 23529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:23.238643 23527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:23.238583 23526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:23.239612 23521 server_base.cc:1034] running on GCE node
I20260926 02:07:23.240409 23521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:23.241628 23521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:23.242913 23521 hybrid_clock.cc:648] HybridClock initialized: now 1790388443242830 us; error 64 us; skew 500 ppm
I20260926 02:07:23.243328 23521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:23 dist-test-slave-03rn krb5kdc[23150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388443, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:23.247193 23521 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:07:23.248785 23521 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:23.249002 23521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:23.249365 23521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:23.253108 23521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "5e7aa08a8a1c4015a7c457e8b3a5bf95"
format_stamp: "Formatted at 2026-09-26 02:07:23 on dist-test-slave-03rn"
server_key: "ecc1707ca6ffecdc2bacc7525eb47859"
server_key_iv: "707f3131409df13ccc87c9935173e39d"
server_key_version: "encryptionkey@0"
I20260926 02:07:23.254032 23521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "5e7aa08a8a1c4015a7c457e8b3a5bf95"
format_stamp: "Formatted at 2026-09-26 02:07:23 on dist-test-slave-03rn"
server_key: "ecc1707ca6ffecdc2bacc7525eb47859"
server_key_iv: "707f3131409df13ccc87c9935173e39d"
server_key_version: "encryptionkey@0"
I20260926 02:07:23.260090 23521 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 02:07:23.264583 23535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:23.266474 23521 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:07:23.266773 23521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "5e7aa08a8a1c4015a7c457e8b3a5bf95"
format_stamp: "Formatted at 2026-09-26 02:07:23 on dist-test-slave-03rn"
server_key: "ecc1707ca6ffecdc2bacc7525eb47859"
server_key_iv: "707f3131409df13ccc87c9935173e39d"
server_key_version: "encryptionkey@0"
I20260926 02:07:23.267071 23521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:23.292315 23521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:23.297979 23521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:23.298305 23521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:23.299428 23521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:23.300995 23521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:23.301136 23521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:23.301306 23521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:23.301405 23521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:23.310815 23382 heartbeater.cc:503] Master 127.18.159.190:39577 was elected leader, sending a full tablet report...
I20260926 02:07:23.338526 23521 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:42017
I20260926 02:07:23.338583 23647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:42017 every 8 connection(s)
I20260926 02:07:23.339423 23521 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:07:23.343850 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 23521
I20260926 02:07:23.344067 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:07:23.344691 19070 external_mini_cluster.cc:1428] Setting key c6eb5a568cd5c6f60186ed78749e5273
Sep 26 02:07:23 dist-test-slave-03rn krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388443, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:23.363270 23650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:23.342159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:57691 (local address 127.18.159.190:39577)
0926 02:07:23.342620 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:23.342626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:23.343843 (+ 1217us) server_negotiation.cc:407] Connection header received
0926 02:07:23.346349 (+ 2506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:23.346354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:23.346437 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:23.346549 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:23.348547 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.349326 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.351466 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.351972 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.352082 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:23.355960 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:23.355966 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:23.355970 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:23.356016 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:23.359228 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:23.360426 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:23.360432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:23.360435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:23.360594 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:23.361178 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:23.361181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:23.361184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:23.361588 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:23.361751 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:23.362755 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0926 02:07:23.362932 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":221,"threads_started":1}
I20260926 02:07:23.364339 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:23.342780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:39577 (local address 127.18.159.131:57691)
0926 02:07:23.343626 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:23.343692 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:23.346019 (+ 2327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:23.346878 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:23.346895 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:23.347499 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:23.348381 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:23.348402 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.349486 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.349489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:23.351271 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:23.351306 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.352301 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.352308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:23.352559 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:23.353383 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:23.353408 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:23.355702 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:23.359441 (+ 3739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:23.359452 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:23.359474 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:23.360278 (+ 804us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:23.360749 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:23.360756 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:23.360762 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:23.361060 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:23.361754 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:23.361766 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:23.362335 (+ 569us) client_negotiation.cc:769] Sending connection context
0926 02:07:23.362663 (+ 328us) client_negotiation.cc:240] Negotiation successful
0926 02:07:23.363106 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":308,"threads_started":1}
I20260926 02:07:23.366062 23648 heartbeater.cc:347] Connected to a master server at 127.18.159.190:39577
I20260926 02:07:23.366420 23648 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:23.367142 23648 heartbeater.cc:511] Master 127.18.159.190:39577 requested a full tablet report, sending...
I20260926 02:07:23.368849 23194 ts_manager.cc:194] Registered new tserver with Master: 5e7aa08a8a1c4015a7c457e8b3a5bf95 (127.18.159.131:42017)
I20260926 02:07:23.369939 23194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:57691
I20260926 02:07:23.373801 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:07:23.393769 23650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:23.378713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44428 (local address 127.18.159.190:39577)
0926 02:07:23.378897 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:23.378902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:23.379300 (+ 398us) server_negotiation.cc:407] Connection header received
0926 02:07:23.379455 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:23.379459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:23.379528 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:23.379624 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:23.381453 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.382264 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.383697 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.384148 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.384374 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:23.387315 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:23.387323 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:23.387327 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:23.387395 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:23.390927 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:23.391796 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:23.391801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:23.391804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:23.391971 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:23.392463 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:23.392467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:23.392469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:23.392790 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:23.393083 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:23.393350 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 02:07:23.393528 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:07:23.398783 23194 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44428:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:07:23.406138 23194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:07:23.448206 23665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:23.420532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.18.159.131:42017)
0926 02:07:23.438860 (+ 18328us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:23.438873 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:23.438932 (+ 59us) server_negotiation.cc:407] Connection header received
0926 02:07:23.439066 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:23.439075 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:23.439366 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:23.439585 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:23.440474 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.442047 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.444936 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.446624 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.446814 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:23.447036 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:23.447633 (+ 597us) server_negotiation.cc:299] Negotiation successful
0926 02:07:23.447806 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18073,"thread_start_us":17885,"threads_started":1}
I20260926 02:07:23.448701 23661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:23.419389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:46251 (local address 127.0.0.1:42080)
0926 02:07:23.435981 (+ 16592us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:23.436049 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:23.437123 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:23.440900 (+ 3777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:23.440907 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:23.440943 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:23.441276 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:23.441287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.443127 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.443133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:23.445585 (+ 2452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:23.445598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.447780 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.447786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:23.447983 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:23.448037 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 02:07:23.448112 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 02:07:23.448297 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16181,"mutex_wait_us":53,"thread_start_us":16809,"threads_started":1}
I20260926 02:07:23.448313 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:23.419822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:42017 (local address 127.0.0.1:45286)
0926 02:07:23.437982 (+ 18160us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:23.438009 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:23.438156 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:23.439816 (+ 1660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:23.439824 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:23.439865 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:23.440243 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:23.440256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.442192 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.442200 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:23.444756 (+ 2556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:23.444773 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.447183 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.447189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:23.447365 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:23.447403 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 02:07:23.447556 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 02:07:23.447777 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17968,"mutex_wait_us":1,"thread_start_us":19219,"threads_started":1}
I20260926 02:07:23.449244 23662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:23.419608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42080 (local address 127.18.159.130:46251)
0926 02:07:23.440312 (+ 20704us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:23.440323 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:23.440348 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:07:23.440442 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:23.440449 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:23.440672 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:23.440888 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:23.441471 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.442968 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.446074 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.447525 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.448447 (+ 922us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:23.448625 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:23.448713 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 02:07:23.448894 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20508,"thread_start_us":20373,"threads_started":1}
I20260926 02:07:23.449290 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:23.420704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.18.159.129:33849)
0926 02:07:23.438647 (+ 17943us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:23.438656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:23.439749 (+ 1093us) server_negotiation.cc:407] Connection header received
0926 02:07:23.439896 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:23.439908 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:23.440208 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:23.440444 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:23.441128 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.443253 (+ 2125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.445964 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.447525 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.447749 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:23.447935 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:23.448795 (+ 860us) server_negotiation.cc:299] Negotiation successful
0926 02:07:23.448952 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17753,"thread_start_us":17952,"threads_started":1}
I20260926 02:07:23.449584 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:23.420053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:33849 (local address 127.0.0.1:33798)
0926 02:07:23.439388 (+ 19335us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:23.439415 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:23.439590 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:23.440452 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:23.440457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:23.440481 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:23.440915 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:23.440928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.443461 (+ 2533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.443467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:23.445784 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:23.445799 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.448083 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.448091 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:23.448290 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:23.448324 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 02:07:23.448405 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 02:07:23.448549 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19172,"mutex_wait_us":21,"thread_start_us":17535,"threads_started":1}
I20260926 02:07:23.453562 23583 tablet_service.cc:1467] Processing CreateTablet for tablet f294805490d74234851e581b351f2054 (DEFAULT_TABLE table=test-table [id=e902972708964085897270f9e91196d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:23.453577 23317 tablet_service.cc:1467] Processing CreateTablet for tablet f294805490d74234851e581b351f2054 (DEFAULT_TABLE table=test-table [id=e902972708964085897270f9e91196d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:23.453577 23450 tablet_service.cc:1467] Processing CreateTablet for tablet f294805490d74234851e581b351f2054 (DEFAULT_TABLE table=test-table [id=e902972708964085897270f9e91196d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:23.454903 23583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f294805490d74234851e581b351f2054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:23.454975 23450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f294805490d74234851e581b351f2054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:23.454981 23317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f294805490d74234851e581b351f2054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:23.467586 23667 tablet_bootstrap.cc:492] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95: Bootstrap starting.
I20260926 02:07:23.470886 23669 tablet_bootstrap.cc:492] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f: Bootstrap starting.
I20260926 02:07:23.471529 23668 tablet_bootstrap.cc:492] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547: Bootstrap starting.
I20260926 02:07:23.472283 23667 tablet_bootstrap.cc:654] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:23.473830 23667 log.cc:824] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:23.475513 23669 tablet_bootstrap.cc:654] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:23.475509 23668 tablet_bootstrap.cc:654] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:23.476750 23668 log.cc:824] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:23.476840 23669 log.cc:824] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:23.477880 23667 tablet_bootstrap.cc:492] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95: No bootstrap required, opened a new log
I20260926 02:07:23.478360 23667 ts_tablet_manager.cc:1397] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260926 02:07:23.479553 23668 tablet_bootstrap.cc:492] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547: No bootstrap required, opened a new log
I20260926 02:07:23.480191 23668 ts_tablet_manager.cc:1397] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 02:07:23.487752 23669 tablet_bootstrap.cc:492] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f: No bootstrap required, opened a new log
I20260926 02:07:23.488128 23669 ts_tablet_manager.cc:1397] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.003s
I20260926 02:07:23.489213 23668 raft_consensus.cc:348] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975f7013538640789b0b2c73895ea547" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46251 } } peers { permanent_uuid: "5e7aa08a8a1c4015a7c457e8b3a5bf95" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 42017 } } peers { permanent_uuid: "5c850c9f00ce450b9734e660210e5e6f" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33849 } }
I20260926 02:07:23.489661 23668 raft_consensus.cc:374] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:23.489799 23668 raft_consensus.cc:729] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975f7013538640789b0b2c73895ea547, State: Initialized, Role: FOLLOWER
I20260926 02:07:23.490499 23668 consensus_queue.cc:260] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975f7013538640789b0b2c73895ea547" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46251 } } peers { permanent_uuid: "5e7aa08a8a1c4015a7c457e8b3a5bf95" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 42017 } } peers { permanent_uuid: "5c850c9f00ce450b9734e660210e5e6f" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33849 } }
I20260926 02:07:23.491091 23667 raft_consensus.cc:348] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975f7013538640789b0b2c73895ea547" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46251 } } peers { permanent_uuid: "5e7aa08a8a1c4015a7c457e8b3a5bf95" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 42017 } } peers { permanent_uuid: "5c850c9f00ce450b9734e660210e5e6f" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33849 } }
I20260926 02:07:23.491623 23667 raft_consensus.cc:374] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:23.491762 23667 raft_consensus.cc:729] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7aa08a8a1c4015a7c457e8b3a5bf95, State: Initialized, Role: FOLLOWER
I20260926 02:07:23.492601 23667 consensus_queue.cc:260] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975f7013538640789b0b2c73895ea547" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46251 } } peers { permanent_uuid: "5e7aa08a8a1c4015a7c457e8b3a5bf95" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 42017 } } peers { permanent_uuid: "5c850c9f00ce450b9734e660210e5e6f" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33849 } }
I20260926 02:07:23.494002 23648 heartbeater.cc:503] Master 127.18.159.190:39577 was elected leader, sending a full tablet report...
I20260926 02:07:23.494652 23667 ts_tablet_manager.cc:1428] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260926 02:07:23.495635 23515 heartbeater.cc:503] Master 127.18.159.190:39577 was elected leader, sending a full tablet report...
I20260926 02:07:23.496409 23668 ts_tablet_manager.cc:1428] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547: Time spent starting tablet: real 0.016s user 0.001s sys 0.014s
I20260926 02:07:23.501029 23669 raft_consensus.cc:348] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975f7013538640789b0b2c73895ea547" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46251 } } peers { permanent_uuid: "5e7aa08a8a1c4015a7c457e8b3a5bf95" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 42017 } } peers { permanent_uuid: "5c850c9f00ce450b9734e660210e5e6f" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33849 } }
I20260926 02:07:23.501524 23669 raft_consensus.cc:374] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:23.501667 23669 raft_consensus.cc:729] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c850c9f00ce450b9734e660210e5e6f, State: Initialized, Role: FOLLOWER
I20260926 02:07:23.502281 23669 consensus_queue.cc:260] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975f7013538640789b0b2c73895ea547" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46251 } } peers { permanent_uuid: "5e7aa08a8a1c4015a7c457e8b3a5bf95" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 42017 } } peers { permanent_uuid: "5c850c9f00ce450b9734e660210e5e6f" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33849 } }
I20260926 02:07:23.504065 23669 ts_tablet_manager.cc:1428] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f: Time spent starting tablet: real 0.016s user 0.012s sys 0.001s
W20260926 02:07:23.523896 23383 tablet.cc:2369] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:07:23.551800 23516 tablet.cc:2369] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:07:23.591120 23649 tablet.cc:2369] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:07:23.609478 23674 raft_consensus.cc:482] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:07:23.609834 23674 raft_consensus.cc:504] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975f7013538640789b0b2c73895ea547" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46251 } } peers { permanent_uuid: "5e7aa08a8a1c4015a7c457e8b3a5bf95" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 42017 } } peers { permanent_uuid: "5c850c9f00ce450b9734e660210e5e6f" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33849 } }
I20260926 02:07:23.611625 23674 leader_election.cc:288] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975f7013538640789b0b2c73895ea547 (127.18.159.130:46251), 5c850c9f00ce450b9734e660210e5e6f (127.18.159.129:33849)
I20260926 02:07:23.617348 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:23.612250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:55305 (local address 127.18.159.129:33849)
0926 02:07:23.612419 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:23.612427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:23.612449 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:07:23.612553 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:23.612560 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:23.612648 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:23.612756 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:23.613191 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.613937 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.615613 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.616450 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.616622 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:23.616699 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:23.616968 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 02:07:23.617093 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:07:23.617372 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:23.611972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:33849 (local address 127.18.159.131:55305)
0926 02:07:23.612207 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:23.612227 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:23.612359 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:23.612810 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:23.612814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:23.612844 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:23.613026 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:23.613034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.614080 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.614087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:23.615438 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:23.615450 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.616608 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.616614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:23.616803 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:23.616826 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:07:23.616944 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 02:07:23.617081 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 02:07:23.618181 23662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:23.612194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:33463 (local address 127.18.159.130:46251)
0926 02:07:23.612396 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:23.612402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:23.613080 (+ 678us) server_negotiation.cc:407] Connection header received
0926 02:07:23.613274 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:23.613278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:23.613341 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:23.613450 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:23.613949 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.614672 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.616397 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.617180 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.617356 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:23.617426 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:23.617757 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 02:07:23.617916 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:07:23.618777 23676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:23.611992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:46251 (local address 127.18.159.131:33463)
0926 02:07:23.612955 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:23.612969 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:23.613123 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:23.613462 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:23.613466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:23.613488 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:23.613781 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:23.613790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.614821 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.614827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:23.616237 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:23.616248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.617390 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.617396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:23.617553 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:23.617576 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:07:23.618374 (+ 798us) client_negotiation.cc:240] Negotiation successful
0926 02:07:23.618510 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":182,"thread_start_us":411,"threads_started":1}
I20260926 02:07:23.618654 23337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f294805490d74234851e581b351f2054" candidate_uuid: "5e7aa08a8a1c4015a7c457e8b3a5bf95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c850c9f00ce450b9734e660210e5e6f" is_pre_election: true
I20260926 02:07:23.619261 23337 raft_consensus.cc:2454] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e7aa08a8a1c4015a7c457e8b3a5bf95 in term 0.
I20260926 02:07:23.619859 23470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f294805490d74234851e581b351f2054" candidate_uuid: "5e7aa08a8a1c4015a7c457e8b3a5bf95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975f7013538640789b0b2c73895ea547" is_pre_election: true
I20260926 02:07:23.620072 23539 leader_election.cc:302] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c850c9f00ce450b9734e660210e5e6f, 5e7aa08a8a1c4015a7c457e8b3a5bf95; no voters:
I20260926 02:07:23.620553 23674 raft_consensus.cc:2781] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:07:23.620563 23470 raft_consensus.cc:2454] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e7aa08a8a1c4015a7c457e8b3a5bf95 in term 0.
I20260926 02:07:23.620680 23674 raft_consensus.cc:482] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:07:23.620826 23674 raft_consensus.cc:3037] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:23.623127 23674 raft_consensus.cc:504] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975f7013538640789b0b2c73895ea547" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46251 } } peers { permanent_uuid: "5e7aa08a8a1c4015a7c457e8b3a5bf95" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 42017 } } peers { permanent_uuid: "5c850c9f00ce450b9734e660210e5e6f" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33849 } }
I20260926 02:07:23.623775 23674 leader_election.cc:288] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 [CANDIDATE]: Term 1 election: Requested vote from peers 975f7013538640789b0b2c73895ea547 (127.18.159.130:46251), 5c850c9f00ce450b9734e660210e5e6f (127.18.159.129:33849)
I20260926 02:07:23.624277 23470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f294805490d74234851e581b351f2054" candidate_uuid: "5e7aa08a8a1c4015a7c457e8b3a5bf95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975f7013538640789b0b2c73895ea547"
I20260926 02:07:23.624353 23337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f294805490d74234851e581b351f2054" candidate_uuid: "5e7aa08a8a1c4015a7c457e8b3a5bf95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c850c9f00ce450b9734e660210e5e6f"
I20260926 02:07:23.624491 23470 raft_consensus.cc:3037] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:23.624571 23337 raft_consensus.cc:3037] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:23.626972 23470 raft_consensus.cc:2454] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e7aa08a8a1c4015a7c457e8b3a5bf95 in term 1.
I20260926 02:07:23.627470 23536 leader_election.cc:302] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e7aa08a8a1c4015a7c457e8b3a5bf95, 975f7013538640789b0b2c73895ea547; no voters:
I20260926 02:07:23.627840 23674 raft_consensus.cc:2781] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:23.627857 23337 raft_consensus.cc:2454] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e7aa08a8a1c4015a7c457e8b3a5bf95 in term 1.
I20260926 02:07:23.628557 23674 raft_consensus.cc:686] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 [term 1 LEADER]: Becoming Leader. State: Replica: 5e7aa08a8a1c4015a7c457e8b3a5bf95, State: Running, Role: LEADER
I20260926 02:07:23.629101 23674 consensus_queue.cc:237] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975f7013538640789b0b2c73895ea547" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46251 } } peers { permanent_uuid: "5e7aa08a8a1c4015a7c457e8b3a5bf95" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 42017 } } peers { permanent_uuid: "5c850c9f00ce450b9734e660210e5e6f" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33849 } }
I20260926 02:07:23.634630 23194 catalog_manager.cc:5495] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e7aa08a8a1c4015a7c457e8b3a5bf95 (127.18.159.131). New cstate: current_term: 1 leader_uuid: "5e7aa08a8a1c4015a7c457e8b3a5bf95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975f7013538640789b0b2c73895ea547" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e7aa08a8a1c4015a7c457e8b3a5bf95" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 42017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c850c9f00ce450b9734e660210e5e6f" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33849 } health_report { overall_health: UNKNOWN } } }
I20260926 02:07:23.695159 23665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:23.689211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45294 (local address 127.18.159.131:42017)
0926 02:07:23.689454 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:23.689460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:23.689599 (+ 139us) server_negotiation.cc:407] Connection header received
0926 02:07:23.689867 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:23.689872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:23.689953 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:23.690169 (+ 216us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:07:23.690579 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.691329 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.692842 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.693277 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.693429 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:23.693992 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:07:23.694520 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:07:23.694721 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:23.694773 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 02:07:23.694861 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:07:23.704542 23337 raft_consensus.cc:1261] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f [term 1 FOLLOWER]: Refusing update from remote peer 5e7aa08a8a1c4015a7c457e8b3a5bf95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:23.704533 23470 raft_consensus.cc:1261] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547 [term 1 FOLLOWER]: Refusing update from remote peer 5e7aa08a8a1c4015a7c457e8b3a5bf95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:23.705624 23677 consensus_queue.cc:1035] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975f7013538640789b0b2c73895ea547" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 46251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:23.706039 23674 consensus_queue.cc:1035] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c850c9f00ce450b9734e660210e5e6f" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:23.734489 23194 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:23.734730 23194 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:23.738780 23194 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 554a848bab004430a49adc03da9d3877: Sending DeleteTablet for 3 replicas of tablet f294805490d74234851e581b351f2054
I20260926 02:07:23.740211 23317 tablet_service.cc:1514] Processing DeleteTablet for tablet f294805490d74234851e581b351f2054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:23 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:33798
I20260926 02:07:23.740238 23583 tablet_service.cc:1514] Processing DeleteTablet for tablet f294805490d74234851e581b351f2054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:23 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:45286
I20260926 02:07:23.741076 23450 tablet_service.cc:1514] Processing DeleteTablet for tablet f294805490d74234851e581b351f2054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:23 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:42080
I20260926 02:07:23.741254 23690 tablet_replica.cc:330] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f: stopping tablet replica
I20260926 02:07:23.741739 23690 raft_consensus.cc:2229] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:23.742303 23690 raft_consensus.cc:2258] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:23.743470 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 23255
I20260926 02:07:23.743891 23689 tablet_replica.cc:330] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95: stopping tablet replica
I20260926 02:07:23.743950 23691 tablet_replica.cc:330] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547: stopping tablet replica
I20260926 02:07:23.744397 23691 raft_consensus.cc:2229] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:23.744413 23689 raft_consensus.cc:2229] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:07:23.745016 23691 raft_consensus.cc:2258] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:23.745137 23689 raft_consensus.cc:2258] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:23.747651 23691 ts_tablet_manager.cc:1905] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:23.747957 23690 ts_tablet_manager.cc:1905] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:23.750530 23689 ts_tablet_manager.cc:1905] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:23.755286 23691 ts_tablet_manager.cc:1918] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:07:23.755832 23691 log.cc:1196] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/f294805490d74234851e581b351f2054
I20260926 02:07:23.756544 23691 ts_tablet_manager.cc:1939] T f294805490d74234851e581b351f2054 P 975f7013538640789b0b2c73895ea547: Deleting consensus metadata
I20260926 02:07:23.758747 23178 catalog_manager.cc:4841] TS 975f7013538640789b0b2c73895ea547 (127.18.159.130:46251): tablet f294805490d74234851e581b351f2054 (table test-table [id=e902972708964085897270f9e91196d4]) successfully deleted
I20260926 02:07:23.762748 23689 ts_tablet_manager.cc:1918] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:07:23.766337 23689 log.cc:1196] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/f294805490d74234851e581b351f2054
I20260926 02:07:23.767133 23689 ts_tablet_manager.cc:1939] T f294805490d74234851e581b351f2054 P 5e7aa08a8a1c4015a7c457e8b3a5bf95: Deleting consensus metadata
I20260926 02:07:23.769248 23180 catalog_manager.cc:4841] TS 5e7aa08a8a1c4015a7c457e8b3a5bf95 (127.18.159.131:42017): tablet f294805490d74234851e581b351f2054 (table test-table [id=e902972708964085897270f9e91196d4]) successfully deleted
I20260926 02:07:23.770764 23690 ts_tablet_manager.cc:1918] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:07:23.770959 23690 log.cc:1196] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/f294805490d74234851e581b351f2054
I20260926 02:07:23.771543 23690 ts_tablet_manager.cc:1939] T f294805490d74234851e581b351f2054 P 5c850c9f00ce450b9734e660210e5e6f: Deleting consensus metadata
Sep 26 02:07:23 dist-test-slave-03rn krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:07:23.773554 23181 catalog_manager.cc:4841] TS 5c850c9f00ce450b9734e660210e5e6f (127.18.159.129:33849): tablet f294805490d74234851e581b351f2054 (table test-table [id=e902972708964085897270f9e91196d4]) successfully deleted
I20260926 02:07:23.780596 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:23.746043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33802 (local address 127.18.159.129:33849)
0926 02:07:23.746290 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:23.746299 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:23.761267 (+ 14968us) server_negotiation.cc:407] Connection header received
0926 02:07:23.761782 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:23.761789 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:23.761882 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:23.761962 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:23.763778 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.764974 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.766834 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.767559 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.767677 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:23.773518 (+ 5841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:23.773534 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:23.773555 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:23.773610 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:23.777561 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:23.778361 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:23.778372 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:23.778376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:23.778560 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:23.778946 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:23.778953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:23.778957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:23.779440 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:23.779750 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:23.779939 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 02:07:23.780205 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:07:23.781459 23377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:23.872632 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 23255
I20260926 02:07:23.890507 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 23388
Sep 26 02:07:23 dist-test-slave-03rn krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.130@KRBTEST.COM
I20260926 02:07:23.907543 23662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:23.891422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42092 (local address 127.18.159.130:46251)
0926 02:07:23.891593 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:23.891598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:23.891617 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:07:23.891724 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:23.891728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:23.891786 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:23.891868 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:23.893481 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.894242 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.895552 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:23.895949 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:23.896106 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:23.900554 (+ 4448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:23.900576 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:23.900597 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:23.900654 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:23.904586 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:23.905368 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:23.905379 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:23.905383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:23.905561 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:23.905911 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:23.905916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:23.905919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:23.906421 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:23.906717 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:23.906921 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 02:07:23.907151 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:07:23.908360 23510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:23.998873 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 23388
I20260926 02:07:24.016096 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 23521
Sep 26 02:07:24 dist-test-slave-03rn krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.131@KRBTEST.COM
I20260926 02:07:24.035394 23665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:24.017022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45300 (local address 127.18.159.131:42017)
0926 02:07:24.017227 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:24.017233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:24.017256 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:07:24.017367 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:24.017373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:24.017473 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:24.017607 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:24.018906 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:24.020101 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:24.021593 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:24.022319 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:24.022498 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:24.027388 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:24.027407 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:24.027428 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:24.027492 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:24.031754 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:24.032856 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:24.032866 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:24.032870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:24.033078 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:24.033512 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:24.033517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:24.033520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:24.034006 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:24.034361 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:24.034711 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 02:07:24.034956 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 02:07:24.036350 23643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:24.159145 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 23521
I20260926 02:07:24.178357 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 23163
I20260926 02:07:24.179579 23224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:24.287433 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 23163
2026-09-26T02:07:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4503 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T02:07:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:07:24Z Disabled control of system clock
I20260926 02:07:24.357734 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:36831
--webserver_interface=127.18.159.190
--webserver_port=0
--builtin_ntp_servers=127.18.159.148:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:36831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 02:07:24.705034 23700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:24.705359 23700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:24.705443 23700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:24.712884 23700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:07:24.713006 23700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:24.713049 23700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:24.713100 23700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:07:24.713145 23700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:07:24.722774 23700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:36831
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:36831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:24.724092 23700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:24.726282 23700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:24.738919 23705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:24.738991 23706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:24.738957 23708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:24.739912 23700 server_base.cc:1034] running on GCE node
I20260926 02:07:24.740844 23700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:24.742412 23700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:24.743697 23700 hybrid_clock.cc:648] HybridClock initialized: now 1790388444743649 us; error 39 us; skew 500 ppm
I20260926 02:07:24.744145 23700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:07:24.755323 23700 webserver.cc:460] Webserver started at http://127.18.159.190:34419/ using document root <none> and password file <none>
I20260926 02:07:24.756251 23700 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:24.756395 23700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:24.756745 23700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:24.759814 23700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "dd6cda5937684a659a159c9e3c60fdec"
format_stamp: "Formatted at 2026-09-26 02:07:24 on dist-test-slave-03rn"
server_key: "6e63345f03a1a72dd67407956c12259b"
server_key_iv: "6de61262eb0e8b953c20dcd1e433ac64"
server_key_version: "encryptionkey@0"
I20260926 02:07:24.760761 23700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "dd6cda5937684a659a159c9e3c60fdec"
format_stamp: "Formatted at 2026-09-26 02:07:24 on dist-test-slave-03rn"
server_key: "6e63345f03a1a72dd67407956c12259b"
server_key_iv: "6de61262eb0e8b953c20dcd1e433ac64"
server_key_version: "encryptionkey@0"
I20260926 02:07:24.766314 23700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:07:24.770645 23714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:24.772331 23700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:07:24.772588 23700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "dd6cda5937684a659a159c9e3c60fdec"
format_stamp: "Formatted at 2026-09-26 02:07:24 on dist-test-slave-03rn"
server_key: "6e63345f03a1a72dd67407956c12259b"
server_key_iv: "6de61262eb0e8b953c20dcd1e433ac64"
server_key_version: "encryptionkey@0"
I20260926 02:07:24.772867 23700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:24.779084 23700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:24.779947 23700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:24.780375 23700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:24.809238 23700 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:36831
I20260926 02:07:24.809257 23765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:36831 every 8 connection(s)
I20260926 02:07:24.811115 23700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:07:24.811775 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 23700
I20260926 02:07:24.811977 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:07:24.812908 19070 external_mini_cluster.cc:1428] Setting key 44491e75298b8d07fc5e2dbf46380fb1
I20260926 02:07:24.816015 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:24.824302 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:24.815297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47320 (local address 127.18.159.190:36831)
0926 02:07:24.816805 (+ 1508us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:24.816820 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:24.816883 (+ 63us) server_negotiation.cc:407] Connection header received
0926 02:07:24.819135 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:24.819168 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:24.819818 (+ 650us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:24.820437 (+ 619us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:24.821729 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:24.821739 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:24.821760 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:07:24.821784 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:24.821907 (+ 123us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:07:24.821937 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:24.822205 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:24.822364 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 02:07:24.822728 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":388,"threads_started":1}
I20260926 02:07:24.831384 23766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec: Bootstrap starting.
I20260926 02:07:24.835227 23766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:24.836448 23766 log.cc:824] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:24.839576 23766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec: No bootstrap required, opened a new log
I20260926 02:07:24.847266 23766 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6cda5937684a659a159c9e3c60fdec" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36831 } }
I20260926 02:07:24.847684 23766 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:24.847765 23766 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6cda5937684a659a159c9e3c60fdec, State: Initialized, Role: FOLLOWER
I20260926 02:07:24.848495 23766 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6cda5937684a659a159c9e3c60fdec" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36831 } }
I20260926 02:07:24.848811 23766 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:07:24.848976 23766 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:07:24.849191 23766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:24.851547 23766 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6cda5937684a659a159c9e3c60fdec" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36831 } }
I20260926 02:07:24.852049 23766 leader_election.cc:302] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd6cda5937684a659a159c9e3c60fdec; no voters:
I20260926 02:07:24.852727 23766 leader_election.cc:288] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:07:24.852876 23771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:24.853595 23771 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec [term 1 LEADER]: Becoming Leader. State: Replica: dd6cda5937684a659a159c9e3c60fdec, State: Running, Role: LEADER
I20260926 02:07:24.854126 23771 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6cda5937684a659a159c9e3c60fdec" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36831 } }
I20260926 02:07:24.854938 23766 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:07:24.856966 23773 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd6cda5937684a659a159c9e3c60fdec. Latest consensus state: current_term: 1 leader_uuid: "dd6cda5937684a659a159c9e3c60fdec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6cda5937684a659a159c9e3c60fdec" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36831 } } }
I20260926 02:07:24.857419 23773 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:24.858055 23772 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd6cda5937684a659a159c9e3c60fdec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6cda5937684a659a159c9e3c60fdec" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36831 } } }
I20260926 02:07:24.858443 23772 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:24.859131 23779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:07:24.863839 23779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:07:24.874462 23779 catalog_manager.cc:1347] Generated new cluster ID: 0ad4b79d491b426fac4cef553e40471a
I20260926 02:07:24.874622 23779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:07:24.891839 23779 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:07:24.892414 23779 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:07:24.900599 23779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec: Generated new TSK 0
I20260926 02:07:24.901691 23779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:07:24.916487 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:36831
--builtin_ntp_servers=127.18.159.148:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 02:07:25.264850 23790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:25.265159 23790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:25.265313 23790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:25.272974 23790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:25.273096 23790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:25.273274 23790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:07:25.283089 23790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:36831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:25.284618 23790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:25.286623 23790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:25.300933 23795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:25.300973 23798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:25.300905 23796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:25.302177 23790 server_base.cc:1034] running on GCE node
I20260926 02:07:25.302883 23790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:25.303835 23790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:25.305055 23790 hybrid_clock.cc:648] HybridClock initialized: now 1790388445305008 us; error 37 us; skew 500 ppm
I20260926 02:07:25.305514 23790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:07:25.307865 23790 webserver.cc:460] Webserver started at http://127.18.159.129:45551/ using document root <none> and password file <none>
I20260926 02:07:25.308661 23790 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:25.308749 23790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:25.309121 23790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:25.311995 23790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "d6270cf503824b7eaca6b3a00a95a4ed"
format_stamp: "Formatted at 2026-09-26 02:07:25 on dist-test-slave-03rn"
server_key: "b3a25b6964e96a88a9b74aad7e367df3"
server_key_iv: "320f81187a12e354c8b8b21866c1942e"
server_key_version: "encryptionkey@0"
I20260926 02:07:25.312896 23790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "d6270cf503824b7eaca6b3a00a95a4ed"
format_stamp: "Formatted at 2026-09-26 02:07:25 on dist-test-slave-03rn"
server_key: "b3a25b6964e96a88a9b74aad7e367df3"
server_key_iv: "320f81187a12e354c8b8b21866c1942e"
server_key_version: "encryptionkey@0"
I20260926 02:07:25.318204 23790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:07:25.321827 23804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:25.323629 23790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 02:07:25.323813 23790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "d6270cf503824b7eaca6b3a00a95a4ed"
format_stamp: "Formatted at 2026-09-26 02:07:25 on dist-test-slave-03rn"
server_key: "b3a25b6964e96a88a9b74aad7e367df3"
server_key_iv: "320f81187a12e354c8b8b21866c1942e"
server_key_version: "encryptionkey@0"
I20260926 02:07:25.324062 23790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:25.330413 23790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:25.331326 23790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:25.331606 23790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:25.332672 23790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:25.334124 23790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:25.334210 23790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:25.334359 23790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:25.334451 23790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:25.368052 23790 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:40175
I20260926 02:07:25.368103 23916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:40175 every 8 connection(s)
I20260926 02:07:25.369506 23790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:07:25.378085 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:25.372178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:45673 (local address 127.18.159.190:36831)
0926 02:07:25.372565 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:25.372570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:25.373938 (+ 1368us) server_negotiation.cc:407] Connection header received
0926 02:07:25.375657 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:25.375663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:25.375732 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:25.375859 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:25.377268 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:25.377274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:25.377279 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:07:25.377298 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:25.377345 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:07:25.377356 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:25.377484 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:25.377691 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 02:07:25.377760 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":209,"threads_started":1}
I20260926 02:07:25.379042 23920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:25.372859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:36831 (local address 127.18.159.129:45673)
0926 02:07:25.373769 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:25.373811 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:25.375353 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:25.376100 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:25.376109 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:25.376271 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:25.376974 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:07:25.376976 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:25.377013 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:07:25.377102 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:25.377460 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:25.377472 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:25.377480 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 02:07:25.377799 (+ 319us) client_negotiation.cc:240] Negotiation successful
0926 02:07:25.378082 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":267,"threads_started":1}
I20260926 02:07:25.379854 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 23790
I20260926 02:07:25.380046 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:07:25.380674 23917 heartbeater.cc:347] Connected to a master server at 127.18.159.190:36831
I20260926 02:07:25.380769 19070 external_mini_cluster.cc:1428] Setting key 998871434ec340a2839d6087541c57d9
I20260926 02:07:25.381031 23917 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:25.381696 23917 heartbeater.cc:511] Master 127.18.159.190:36831 requested a full tablet report, sending...
I20260926 02:07:25.383837 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:36831
--builtin_ntp_servers=127.18.159.148:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 02:07:25.384049 23731 ts_manager.cc:194] Registered new tserver with Master: d6270cf503824b7eaca6b3a00a95a4ed (127.18.159.129:40175)
W20260926 02:07:25.720314 23921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:25.720690 23921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:25.720875 23921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:25.728901 23921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:25.729068 23921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:25.729256 23921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:07:25.738791 23921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:36831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:25.740334 23921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:25.742147 23921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:25.756188 23927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:25.756453 23926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:25.756141 23929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:25.757207 23921 server_base.cc:1034] running on GCE node
I20260926 02:07:25.758083 23921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:25.759203 23921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:25.760406 23921 hybrid_clock.cc:648] HybridClock initialized: now 1790388445760340 us; error 61 us; skew 500 ppm
I20260926 02:07:25.760782 23921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:07:25.763512 23921 webserver.cc:460] Webserver started at http://127.18.159.130:35773/ using document root <none> and password file <none>
I20260926 02:07:25.764391 23921 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:25.764489 23921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:25.764822 23921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:25.767869 23921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "7790bde38a714cd5aedb4c76eced0dd8"
format_stamp: "Formatted at 2026-09-26 02:07:25 on dist-test-slave-03rn"
server_key: "b34c282af9ba27403b60d417b1a9e947"
server_key_iv: "e3f1dfb06ad4fabb835a470881f6ab1b"
server_key_version: "encryptionkey@0"
I20260926 02:07:25.768885 23921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "7790bde38a714cd5aedb4c76eced0dd8"
format_stamp: "Formatted at 2026-09-26 02:07:25 on dist-test-slave-03rn"
server_key: "b34c282af9ba27403b60d417b1a9e947"
server_key_iv: "e3f1dfb06ad4fabb835a470881f6ab1b"
server_key_version: "encryptionkey@0"
I20260926 02:07:25.774539 23921 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:07:25.778640 23935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:25.780263 23921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:07:25.780459 23921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "7790bde38a714cd5aedb4c76eced0dd8"
format_stamp: "Formatted at 2026-09-26 02:07:25 on dist-test-slave-03rn"
server_key: "b34c282af9ba27403b60d417b1a9e947"
server_key_iv: "e3f1dfb06ad4fabb835a470881f6ab1b"
server_key_version: "encryptionkey@0"
I20260926 02:07:25.780696 23921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:25.786871 23921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:25.787736 23921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:25.788020 23921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:25.789042 23921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:25.790619 23921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:25.790705 23921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:25.790855 23921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:25.790925 23921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:25.828568 23921 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:38841
I20260926 02:07:25.828665 24047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:38841 every 8 connection(s)
I20260926 02:07:25.830044 23921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:07:25.835788 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 23921
I20260926 02:07:25.836067 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:07:25.836566 19070 external_mini_cluster.cc:1428] Setting key 99660200d3900d6a114afe3d9b83c36d
I20260926 02:07:25.839848 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:36831
--builtin_ntp_servers=127.18.159.148:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 02:07:25.840356 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:25.832814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:42401 (local address 127.18.159.190:36831)
0926 02:07:25.833011 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:25.833016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:25.834550 (+ 1534us) server_negotiation.cc:407] Connection header received
0926 02:07:25.837202 (+ 2652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:25.837208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:25.837284 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:25.837429 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:25.839458 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:25.839463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:25.839467 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:07:25.839488 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:25.839531 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:07:25.839544 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:25.839693 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:25.839952 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 02:07:25.840029 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:07:25.841630 24050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:25.833400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:36831 (local address 127.18.159.130:42401)
0926 02:07:25.834358 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:25.834395 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:25.836560 (+ 2165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:25.837758 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:25.837772 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:25.837943 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:25.838901 (+ 958us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:07:25.838905 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:25.838965 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:07:25.839197 (+ 232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:25.839674 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:25.839686 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:25.839696 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 02:07:25.840072 (+ 376us) client_negotiation.cc:240] Negotiation successful
0926 02:07:25.840447 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":283,"threads_started":1}
I20260926 02:07:25.843756 24048 heartbeater.cc:347] Connected to a master server at 127.18.159.190:36831
I20260926 02:07:25.844135 24048 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:25.844885 24048 heartbeater.cc:511] Master 127.18.159.190:36831 requested a full tablet report, sending...
I20260926 02:07:25.846719 23731 ts_manager.cc:194] Registered new tserver with Master: 7790bde38a714cd5aedb4c76eced0dd8 (127.18.159.130:38841)
W20260926 02:07:26.192847 24051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:26.193199 24051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:26.193358 24051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:26.200855 24051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:26.201015 24051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:26.201203 24051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:07:26.210626 24051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:36831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:26.212044 24051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:26.213899 24051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:26.227663 24057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:26.227645 24056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:26.227946 24059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:26.228636 24051 server_base.cc:1034] running on GCE node
I20260926 02:07:26.229346 24051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:26.230341 24051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:26.231578 24051 hybrid_clock.cc:648] HybridClock initialized: now 1790388446231499 us; error 68 us; skew 500 ppm
I20260926 02:07:26.231961 24051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:07:26.234382 24051 webserver.cc:460] Webserver started at http://127.18.159.131:42309/ using document root <none> and password file <none>
I20260926 02:07:26.235164 24051 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:26.235286 24051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:26.235643 24051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:26.238786 24051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "dba501dd55c748d6863bbf790f3452a1"
format_stamp: "Formatted at 2026-09-26 02:07:26 on dist-test-slave-03rn"
server_key: "9ba71d63f0e28370f32b6a8b74a90ee5"
server_key_iv: "64cef606d4dc4b24ea59df4b99a91d37"
server_key_version: "encryptionkey@0"
I20260926 02:07:26.239739 24051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "dba501dd55c748d6863bbf790f3452a1"
format_stamp: "Formatted at 2026-09-26 02:07:26 on dist-test-slave-03rn"
server_key: "9ba71d63f0e28370f32b6a8b74a90ee5"
server_key_iv: "64cef606d4dc4b24ea59df4b99a91d37"
server_key_version: "encryptionkey@0"
I20260926 02:07:26.245952 24051 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 02:07:26.250222 24065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:26.252035 24051 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 02:07:26.252364 24051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "dba501dd55c748d6863bbf790f3452a1"
format_stamp: "Formatted at 2026-09-26 02:07:26 on dist-test-slave-03rn"
server_key: "9ba71d63f0e28370f32b6a8b74a90ee5"
server_key_iv: "64cef606d4dc4b24ea59df4b99a91d37"
server_key_version: "encryptionkey@0"
I20260926 02:07:26.252645 24051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:26.259230 24051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:26.260229 24051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:26.260617 24051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:26.261719 24051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:26.263273 24051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:26.263419 24051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:26.263574 24051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:26.263664 24051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:26.299849 24051 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:33109
I20260926 02:07:26.299928 24177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:33109 every 8 connection(s)
I20260926 02:07:26.301350 24051 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:07:26.303735 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 24051
I20260926 02:07:26.303908 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:07:26.304466 19070 external_mini_cluster.cc:1428] Setting key b18d3749dac8a95ad90140a15e8324cf
I20260926 02:07:26.310796 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.304699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:56587 (local address 127.18.159.190:36831)
0926 02:07:26.304934 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:26.304941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:26.306831 (+ 1890us) server_negotiation.cc:407] Connection header received
0926 02:07:26.308858 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:26.308862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:26.308919 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:26.309005 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:26.310221 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:26.310225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:26.310228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:07:26.310242 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:26.310270 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:07:26.310281 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:26.310368 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:26.310539 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 02:07:26.310583 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:07:26.311755 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.305379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:36831 (local address 127.18.159.131:56587)
0926 02:07:26.306632 (+ 1253us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:26.306700 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:26.308596 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:26.309210 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:26.309218 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:26.309321 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:26.309946 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:07:26.309948 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:26.309986 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:07:26.310072 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:26.310358 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:26.310363 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:26.310367 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 02:07:26.310572 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 02:07:26.310775 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":412,"threads_started":1}
I20260926 02:07:26.313200 24178 heartbeater.cc:347] Connected to a master server at 127.18.159.190:36831
I20260926 02:07:26.313546 24178 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:26.314236 24178 heartbeater.cc:511] Master 127.18.159.190:36831 requested a full tablet report, sending...
I20260926 02:07:26.315699 23731 ts_manager.cc:194] Registered new tserver with Master: dba501dd55c748d6863bbf790f3452a1 (127.18.159.131:33109)
I20260926 02:07:26.319984 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:07:26.327028 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.324517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47322 (local address 127.18.159.190:36831)
0926 02:07:26.324691 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:26.324696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:26.325302 (+ 606us) server_negotiation.cc:407] Connection header received
0926 02:07:26.325444 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:26.325448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:26.325501 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:26.325608 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:26.326494 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:26.326498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:26.326501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:07:26.326532 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:26.326561 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:07:26.326573 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:26.326676 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:26.326769 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 02:07:26.326816 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:07:26.331519 23731 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:07:26.338516 23731 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:07:26.356858 24190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.351040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:38841 (local address 127.0.0.1:48416)
0926 02:07:26.351712 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:26.351748 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:26.351907 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:26.353801 (+ 1894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:26.353813 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:26.353945 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:26.355045 (+ 1100us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:07:26.355051 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:26.355099 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:07:26.355207 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:26.355857 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:26.355872 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:26.355884 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:07:26.356093 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 02:07:26.356230 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":295,"threads_started":1}
I20260926 02:07:26.357926 24191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.351141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.18.159.130:38841)
0926 02:07:26.351771 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:26.351780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:26.352044 (+ 264us) server_negotiation.cc:407] Connection header received
0926 02:07:26.352119 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:26.352126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:26.352421 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:26.352592 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:26.355470 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:26.355484 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:26.355505 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:07:26.355540 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:26.355619 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:07:26.355645 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:26.355795 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:26.357308 (+ 1513us) server_negotiation.cc:299] Negotiation successful
0926 02:07:26.357462 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":378,"threads_started":1}
I20260926 02:07:26.363395 24190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.352222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40175 (local address 127.0.0.1:43238)
0926 02:07:26.359260 (+ 7038us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:26.359281 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:26.359351 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:26.360881 (+ 1530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:26.360886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:26.360911 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:26.361534 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:07:26.361537 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:26.361554 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:07:26.361618 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:26.362920 (+ 1302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:26.362927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:26.362940 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 02:07:26.363062 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 02:07:26.363116 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6916,"thread_start_us":2700,"threads_started":1}
I20260926 02:07:26.364035 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.352118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:33109 (local address 127.0.0.1:58438)
0926 02:07:26.358776 (+ 6658us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:26.358800 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:26.358964 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:26.361621 (+ 2657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:26.361626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:26.361650 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:26.362793 (+ 1143us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:07:26.362797 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:26.362813 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:07:26.362862 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:26.363393 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:26.363398 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:26.363402 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 02:07:26.363544 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 02:07:26.363591 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6448,"thread_start_us":7857,"threads_started":1}
I20260926 02:07:26.364801 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.352654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58438 (local address 127.18.159.131:33109)
0926 02:07:26.360684 (+ 8030us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:26.360701 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:26.360736 (+ 35us) server_negotiation.cc:407] Connection header received
0926 02:07:26.360805 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:26.360813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:26.361078 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:26.361644 (+ 566us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:26.363059 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:26.363067 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:26.363079 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:07:26.363099 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:26.363156 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:07:26.363175 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:26.363333 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:26.364300 (+ 967us) server_negotiation.cc:299] Negotiation successful
0926 02:07:26.364406 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7868,"thread_start_us":7650,"threads_started":1}
I20260926 02:07:26.364840 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.352711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43238 (local address 127.18.159.129:40175)
0926 02:07:26.360271 (+ 7560us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:26.360286 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:26.360324 (+ 38us) server_negotiation.cc:407] Connection header received
0926 02:07:26.360391 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:26.360399 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:26.360673 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:26.360865 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:26.362649 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:26.362658 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:26.362675 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:07:26.362702 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:26.362770 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:07:26.362792 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:26.362934 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:26.364389 (+ 1455us) server_negotiation.cc:299] Negotiation successful
0926 02:07:26.364476 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7318,"thread_start_us":7183,"threads_started":1}
I20260926 02:07:26.364454 23983 tablet_service.cc:1467] Processing CreateTablet for tablet b813ee5b501741eb991a5353445ff3e0 (DEFAULT_TABLE table=test-table [id=6bcc5afd597842c690420b9f82f4439b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:26.366063 23983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b813ee5b501741eb991a5353445ff3e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:26.369143 23852 tablet_service.cc:1467] Processing CreateTablet for tablet b813ee5b501741eb991a5353445ff3e0 (DEFAULT_TABLE table=test-table [id=6bcc5afd597842c690420b9f82f4439b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:26.370837 23852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b813ee5b501741eb991a5353445ff3e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:26.371650 24113 tablet_service.cc:1467] Processing CreateTablet for tablet b813ee5b501741eb991a5353445ff3e0 (DEFAULT_TABLE table=test-table [id=6bcc5afd597842c690420b9f82f4439b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:26.374135 24113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b813ee5b501741eb991a5353445ff3e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:26.387557 23917 heartbeater.cc:503] Master 127.18.159.190:36831 was elected leader, sending a full tablet report...
I20260926 02:07:26.390046 24196 tablet_bootstrap.cc:492] T b813ee5b501741eb991a5353445ff3e0 P 7790bde38a714cd5aedb4c76eced0dd8: Bootstrap starting.
I20260926 02:07:26.394385 24198 tablet_bootstrap.cc:492] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1: Bootstrap starting.
I20260926 02:07:26.394486 24196 tablet_bootstrap.cc:654] T b813ee5b501741eb991a5353445ff3e0 P 7790bde38a714cd5aedb4c76eced0dd8: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:26.395740 24197 tablet_bootstrap.cc:492] T b813ee5b501741eb991a5353445ff3e0 P d6270cf503824b7eaca6b3a00a95a4ed: Bootstrap starting.
I20260926 02:07:26.395813 24196 log.cc:824] T b813ee5b501741eb991a5353445ff3e0 P 7790bde38a714cd5aedb4c76eced0dd8: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:26.398401 24198 tablet_bootstrap.cc:654] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:26.399399 24197 tablet_bootstrap.cc:654] T b813ee5b501741eb991a5353445ff3e0 P d6270cf503824b7eaca6b3a00a95a4ed: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:26.399762 24198 log.cc:824] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:26.400199 24196 tablet_bootstrap.cc:492] T b813ee5b501741eb991a5353445ff3e0 P 7790bde38a714cd5aedb4c76eced0dd8: No bootstrap required, opened a new log
I20260926 02:07:26.400595 24196 ts_tablet_manager.cc:1397] T b813ee5b501741eb991a5353445ff3e0 P 7790bde38a714cd5aedb4c76eced0dd8: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260926 02:07:26.400662 24197 log.cc:824] T b813ee5b501741eb991a5353445ff3e0 P d6270cf503824b7eaca6b3a00a95a4ed: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:26.404317 24198 tablet_bootstrap.cc:492] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1: No bootstrap required, opened a new log
I20260926 02:07:26.404316 24197 tablet_bootstrap.cc:492] T b813ee5b501741eb991a5353445ff3e0 P d6270cf503824b7eaca6b3a00a95a4ed: No bootstrap required, opened a new log
I20260926 02:07:26.404606 24197 ts_tablet_manager.cc:1397] T b813ee5b501741eb991a5353445ff3e0 P d6270cf503824b7eaca6b3a00a95a4ed: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 02:07:26.404604 24198 ts_tablet_manager.cc:1397] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260926 02:07:26.409511 24196 raft_consensus.cc:348] T b813ee5b501741eb991a5353445ff3e0 P 7790bde38a714cd5aedb4c76eced0dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7790bde38a714cd5aedb4c76eced0dd8" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38841 } } peers { permanent_uuid: "dba501dd55c748d6863bbf790f3452a1" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 33109 } } peers { permanent_uuid: "d6270cf503824b7eaca6b3a00a95a4ed" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40175 } }
I20260926 02:07:26.409999 24196 raft_consensus.cc:374] T b813ee5b501741eb991a5353445ff3e0 P 7790bde38a714cd5aedb4c76eced0dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:26.410100 24196 raft_consensus.cc:729] T b813ee5b501741eb991a5353445ff3e0 P 7790bde38a714cd5aedb4c76eced0dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7790bde38a714cd5aedb4c76eced0dd8, State: Initialized, Role: FOLLOWER
I20260926 02:07:26.410830 24196 consensus_queue.cc:260] T b813ee5b501741eb991a5353445ff3e0 P 7790bde38a714cd5aedb4c76eced0dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7790bde38a714cd5aedb4c76eced0dd8" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38841 } } peers { permanent_uuid: "dba501dd55c748d6863bbf790f3452a1" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 33109 } } peers { permanent_uuid: "d6270cf503824b7eaca6b3a00a95a4ed" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40175 } }
I20260926 02:07:26.412096 24048 heartbeater.cc:503] Master 127.18.159.190:36831 was elected leader, sending a full tablet report...
I20260926 02:07:26.412918 24196 ts_tablet_manager.cc:1428] T b813ee5b501741eb991a5353445ff3e0 P 7790bde38a714cd5aedb4c76eced0dd8: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260926 02:07:26.415815 24198 raft_consensus.cc:348] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7790bde38a714cd5aedb4c76eced0dd8" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38841 } } peers { permanent_uuid: "dba501dd55c748d6863bbf790f3452a1" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 33109 } } peers { permanent_uuid: "d6270cf503824b7eaca6b3a00a95a4ed" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40175 } }
I20260926 02:07:26.415815 24197 raft_consensus.cc:348] T b813ee5b501741eb991a5353445ff3e0 P d6270cf503824b7eaca6b3a00a95a4ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7790bde38a714cd5aedb4c76eced0dd8" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38841 } } peers { permanent_uuid: "dba501dd55c748d6863bbf790f3452a1" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 33109 } } peers { permanent_uuid: "d6270cf503824b7eaca6b3a00a95a4ed" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40175 } }
I20260926 02:07:26.416314 24197 raft_consensus.cc:374] T b813ee5b501741eb991a5353445ff3e0 P d6270cf503824b7eaca6b3a00a95a4ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:26.416314 24198 raft_consensus.cc:374] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:26.416409 24198 raft_consensus.cc:729] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba501dd55c748d6863bbf790f3452a1, State: Initialized, Role: FOLLOWER
I20260926 02:07:26.416409 24197 raft_consensus.cc:729] T b813ee5b501741eb991a5353445ff3e0 P d6270cf503824b7eaca6b3a00a95a4ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6270cf503824b7eaca6b3a00a95a4ed, State: Initialized, Role: FOLLOWER
I20260926 02:07:26.417068 24198 consensus_queue.cc:260] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7790bde38a714cd5aedb4c76eced0dd8" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38841 } } peers { permanent_uuid: "dba501dd55c748d6863bbf790f3452a1" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 33109 } } peers { permanent_uuid: "d6270cf503824b7eaca6b3a00a95a4ed" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40175 } }
I20260926 02:07:26.417071 24197 consensus_queue.cc:260] T b813ee5b501741eb991a5353445ff3e0 P d6270cf503824b7eaca6b3a00a95a4ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7790bde38a714cd5aedb4c76eced0dd8" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38841 } } peers { permanent_uuid: "dba501dd55c748d6863bbf790f3452a1" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 33109 } } peers { permanent_uuid: "d6270cf503824b7eaca6b3a00a95a4ed" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40175 } }
I20260926 02:07:26.418758 24178 heartbeater.cc:503] Master 127.18.159.190:36831 was elected leader, sending a full tablet report...
I20260926 02:07:26.419127 24198 ts_tablet_manager.cc:1428] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260926 02:07:26.420259 24197 ts_tablet_manager.cc:1428] T b813ee5b501741eb991a5353445ff3e0 P d6270cf503824b7eaca6b3a00a95a4ed: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260926 02:07:26.428377 24203 raft_consensus.cc:482] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:07:26.428695 24203 raft_consensus.cc:504] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7790bde38a714cd5aedb4c76eced0dd8" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38841 } } peers { permanent_uuid: "dba501dd55c748d6863bbf790f3452a1" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 33109 } } peers { permanent_uuid: "d6270cf503824b7eaca6b3a00a95a4ed" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40175 } }
I20260926 02:07:26.430477 24203 leader_election.cc:288] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7790bde38a714cd5aedb4c76eced0dd8 (127.18.159.130:38841), d6270cf503824b7eaca6b3a00a95a4ed (127.18.159.129:40175)
I20260926 02:07:26.433215 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.430859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:38841 (local address 127.18.159.131:58439)
0926 02:07:26.431057 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:26.431079 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:26.431186 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:26.431667 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:26.431674 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:26.431708 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:26.432412 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:07:26.432415 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:26.432432 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:07:26.432473 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:26.432796 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:26.432801 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:26.432805 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 02:07:26.432898 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 02:07:26.432950 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 02:07:26.433218 24191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.430854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:58439 (local address 127.18.159.130:38841)
0926 02:07:26.431208 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:26.431214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:26.431234 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:07:26.431299 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:26.431304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:26.431373 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:26.431495 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:26.432632 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:26.432645 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:26.432648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:07:26.432663 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:26.432691 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:07:26.432702 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:26.432824 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:26.432903 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 02:07:26.432958 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234}
I20260926 02:07:26.433890 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.430871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:37985 (local address 127.18.159.129:40175)
0926 02:07:26.431055 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:26.431061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:26.431939 (+ 878us) server_negotiation.cc:407] Connection header received
0926 02:07:26.432053 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:26.432058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:26.432113 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:26.432260 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:26.433199 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:26.433205 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:26.433209 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:07:26.433227 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:26.433264 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:07:26.433296 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:26.433415 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:26.433609 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 02:07:26.433669 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 02:07:26.434168 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.431190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40175 (local address 127.18.159.131:37985)
0926 02:07:26.431801 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:26.431814 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:26.431937 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:26.432276 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:26.432281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:26.432304 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:26.433013 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:07:26.433015 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:26.433027 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:07:26.433059 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:26.433465 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:26.433470 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:26.433474 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 02:07:26.433606 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 02:07:26.433651 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":335,"threads_started":1}
I20260926 02:07:26.434324 24003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b813ee5b501741eb991a5353445ff3e0" candidate_uuid: "dba501dd55c748d6863bbf790f3452a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7790bde38a714cd5aedb4c76eced0dd8" is_pre_election: true
I20260926 02:07:26.434906 24003 raft_consensus.cc:2454] T b813ee5b501741eb991a5353445ff3e0 P 7790bde38a714cd5aedb4c76eced0dd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba501dd55c748d6863bbf790f3452a1 in term 0.
I20260926 02:07:26.435190 23872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b813ee5b501741eb991a5353445ff3e0" candidate_uuid: "dba501dd55c748d6863bbf790f3452a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6270cf503824b7eaca6b3a00a95a4ed" is_pre_election: true
I20260926 02:07:26.435688 24069 leader_election.cc:302] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7790bde38a714cd5aedb4c76eced0dd8, dba501dd55c748d6863bbf790f3452a1; no voters:
I20260926 02:07:26.435737 23872 raft_consensus.cc:2454] T b813ee5b501741eb991a5353445ff3e0 P d6270cf503824b7eaca6b3a00a95a4ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba501dd55c748d6863bbf790f3452a1 in term 0.
I20260926 02:07:26.436076 24203 raft_consensus.cc:2781] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:07:26.436264 24203 raft_consensus.cc:482] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:07:26.436348 24203 raft_consensus.cc:3037] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:26.438490 24203 raft_consensus.cc:504] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7790bde38a714cd5aedb4c76eced0dd8" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38841 } } peers { permanent_uuid: "dba501dd55c748d6863bbf790f3452a1" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 33109 } } peers { permanent_uuid: "d6270cf503824b7eaca6b3a00a95a4ed" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40175 } }
I20260926 02:07:26.439169 24203 leader_election.cc:288] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 [CANDIDATE]: Term 1 election: Requested vote from peers 7790bde38a714cd5aedb4c76eced0dd8 (127.18.159.130:38841), d6270cf503824b7eaca6b3a00a95a4ed (127.18.159.129:40175)
I20260926 02:07:26.439687 24003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b813ee5b501741eb991a5353445ff3e0" candidate_uuid: "dba501dd55c748d6863bbf790f3452a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7790bde38a714cd5aedb4c76eced0dd8"
I20260926 02:07:26.439711 23872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b813ee5b501741eb991a5353445ff3e0" candidate_uuid: "dba501dd55c748d6863bbf790f3452a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6270cf503824b7eaca6b3a00a95a4ed"
I20260926 02:07:26.439899 24003 raft_consensus.cc:3037] T b813ee5b501741eb991a5353445ff3e0 P 7790bde38a714cd5aedb4c76eced0dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:26.439908 23872 raft_consensus.cc:3037] T b813ee5b501741eb991a5353445ff3e0 P d6270cf503824b7eaca6b3a00a95a4ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:26.442186 23872 raft_consensus.cc:2454] T b813ee5b501741eb991a5353445ff3e0 P d6270cf503824b7eaca6b3a00a95a4ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba501dd55c748d6863bbf790f3452a1 in term 1.
I20260926 02:07:26.442188 24003 raft_consensus.cc:2454] T b813ee5b501741eb991a5353445ff3e0 P 7790bde38a714cd5aedb4c76eced0dd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba501dd55c748d6863bbf790f3452a1 in term 1.
I20260926 02:07:26.442723 24066 leader_election.cc:302] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6270cf503824b7eaca6b3a00a95a4ed, dba501dd55c748d6863bbf790f3452a1; no voters:
I20260926 02:07:26.443097 24203 raft_consensus.cc:2781] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:26.443531 24203 raft_consensus.cc:686] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 [term 1 LEADER]: Becoming Leader. State: Replica: dba501dd55c748d6863bbf790f3452a1, State: Running, Role: LEADER
I20260926 02:07:26.444023 24203 consensus_queue.cc:237] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7790bde38a714cd5aedb4c76eced0dd8" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38841 } } peers { permanent_uuid: "dba501dd55c748d6863bbf790f3452a1" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 33109 } } peers { permanent_uuid: "d6270cf503824b7eaca6b3a00a95a4ed" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40175 } }
I20260926 02:07:26.449728 23731 catalog_manager.cc:5495] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to dba501dd55c748d6863bbf790f3452a1 (127.18.159.131). New cstate: current_term: 1 leader_uuid: "dba501dd55c748d6863bbf790f3452a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7790bde38a714cd5aedb4c76eced0dd8" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba501dd55c748d6863bbf790f3452a1" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 33109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6270cf503824b7eaca6b3a00a95a4ed" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40175 } health_report { overall_health: UNKNOWN } } }
I20260926 02:07:26.464488 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.462188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.18.159.131:33109)
0926 02:07:26.462393 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:26.462399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:26.462516 (+ 117us) server_negotiation.cc:407] Connection header received
0926 02:07:26.462789 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:26.462793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:26.462872 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:26.462982 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:26.463857 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:26.463863 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:26.463867 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:07:26.463891 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:26.463922 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:07:26.463933 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:26.464078 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:26.464127 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 02:07:26.464221 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:07:26.472719 24003 raft_consensus.cc:1261] T b813ee5b501741eb991a5353445ff3e0 P 7790bde38a714cd5aedb4c76eced0dd8 [term 1 FOLLOWER]: Refusing update from remote peer dba501dd55c748d6863bbf790f3452a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:26.472713 23872 raft_consensus.cc:1261] T b813ee5b501741eb991a5353445ff3e0 P d6270cf503824b7eaca6b3a00a95a4ed [term 1 FOLLOWER]: Refusing update from remote peer dba501dd55c748d6863bbf790f3452a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:26.473708 24203 consensus_queue.cc:1035] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6270cf503824b7eaca6b3a00a95a4ed" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:26.473990 24206 consensus_queue.cc:1035] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7790bde38a714cd5aedb4c76eced0dd8" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:26.483515 24209 mvcc.cc:205] Tried to move back new op lower bound from 7333431076742275072 to 7333431076639539200. Current Snapshot: MvccSnapshot[applied={T|T < 7333431076742275072}]
I20260926 02:07:26.484589 24211 mvcc.cc:205] Tried to move back new op lower bound from 7333431076742275072 to 7333431076639539200. Current Snapshot: MvccSnapshot[applied={T|T < 7333431076742275072}]
I20260926 02:07:26.492905 24210 mvcc.cc:205] Tried to move back new op lower bound from 7333431076742275072 to 7333431076639539200. Current Snapshot: MvccSnapshot[applied={T|T < 7333431076742275072}]
I20260926 02:07:26.503285 23730 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:26.503598 23730 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:26.507305 23730 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd6cda5937684a659a159c9e3c60fdec: Sending DeleteTablet for 3 replicas of tablet b813ee5b501741eb991a5353445ff3e0
I20260926 02:07:26.508617 23852 tablet_service.cc:1514] Processing DeleteTablet for tablet b813ee5b501741eb991a5353445ff3e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:26 UTC) from {username='slave'} at 127.0.0.1:43238
I20260926 02:07:26.508654 23983 tablet_service.cc:1514] Processing DeleteTablet for tablet b813ee5b501741eb991a5353445ff3e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:26 UTC) from {username='slave'} at 127.0.0.1:48416
I20260926 02:07:26.508672 24113 tablet_service.cc:1514] Processing DeleteTablet for tablet b813ee5b501741eb991a5353445ff3e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:26 UTC) from {username='slave'} at 127.0.0.1:58438
I20260926 02:07:26.510349 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 23790
I20260926 02:07:26.521241 24219 tablet_replica.cc:330] T b813ee5b501741eb991a5353445ff3e0 P d6270cf503824b7eaca6b3a00a95a4ed: stopping tablet replica
I20260926 02:07:26.522073 24219 raft_consensus.cc:2229] T b813ee5b501741eb991a5353445ff3e0 P d6270cf503824b7eaca6b3a00a95a4ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:26.522898 24219 raft_consensus.cc:2258] T b813ee5b501741eb991a5353445ff3e0 P d6270cf503824b7eaca6b3a00a95a4ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:26.524185 24218 tablet_replica.cc:330] T b813ee5b501741eb991a5353445ff3e0 P 7790bde38a714cd5aedb4c76eced0dd8: stopping tablet replica
I20260926 02:07:26.525200 24220 tablet_replica.cc:330] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1: stopping tablet replica
I20260926 02:07:26.525499 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.512111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43240 (local address 127.18.159.129:40175)
0926 02:07:26.512365 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:26.512374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:26.522633 (+ 10259us) server_negotiation.cc:407] Connection header received
0926 02:07:26.523164 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:26.523172 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:26.523303 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:26.523384 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:26.524665 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:26.524674 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:26.524679 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:07:26.524701 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:26.524739 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:07:26.524757 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:26.524862 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:26.525136 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 02:07:26.525201 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 02:07:26.525674 24220 raft_consensus.cc:2229] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:07:26.526113 24218 raft_consensus.cc:2229] T b813ee5b501741eb991a5353445ff3e0 P 7790bde38a714cd5aedb4c76eced0dd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:26.526482 23912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:26.527315 24220 raft_consensus.cc:2258] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:26.527524 24218 raft_consensus.cc:2258] T b813ee5b501741eb991a5353445ff3e0 P 7790bde38a714cd5aedb4c76eced0dd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:26.529871 24220 ts_tablet_manager.cc:1905] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:26.530131 24218 ts_tablet_manager.cc:1905] T b813ee5b501741eb991a5353445ff3e0 P 7790bde38a714cd5aedb4c76eced0dd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:26.537678 24220 ts_tablet_manager.cc:1918] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:07:26.537875 24220 log.cc:1196] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/b813ee5b501741eb991a5353445ff3e0
I20260926 02:07:26.538475 24220 ts_tablet_manager.cc:1939] T b813ee5b501741eb991a5353445ff3e0 P dba501dd55c748d6863bbf790f3452a1: Deleting consensus metadata
I20260926 02:07:26.538481 24218 ts_tablet_manager.cc:1918] T b813ee5b501741eb991a5353445ff3e0 P 7790bde38a714cd5aedb4c76eced0dd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:07:26.538630 24218 log.cc:1196] T b813ee5b501741eb991a5353445ff3e0 P 7790bde38a714cd5aedb4c76eced0dd8: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/b813ee5b501741eb991a5353445ff3e0
I20260926 02:07:26.539187 24218 ts_tablet_manager.cc:1939] T b813ee5b501741eb991a5353445ff3e0 P 7790bde38a714cd5aedb4c76eced0dd8: Deleting consensus metadata
I20260926 02:07:26.540634 23718 catalog_manager.cc:4841] TS dba501dd55c748d6863bbf790f3452a1 (127.18.159.131:33109): tablet b813ee5b501741eb991a5353445ff3e0 (table test-table [id=6bcc5afd597842c690420b9f82f4439b]) successfully deleted
I20260926 02:07:26.541035 23718 catalog_manager.cc:4841] TS 7790bde38a714cd5aedb4c76eced0dd8 (127.18.159.130:38841): tablet b813ee5b501741eb991a5353445ff3e0 (table test-table [id=6bcc5afd597842c690420b9f82f4439b]) successfully deleted
I20260926 02:07:26.526806 24219 ts_tablet_manager.cc:1905] T b813ee5b501741eb991a5353445ff3e0 P d6270cf503824b7eaca6b3a00a95a4ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:26.624413 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 23790
I20260926 02:07:26.642297 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 23921
W20260926 02:07:26.642357 23715 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.159.129:40175 (error 108)
I20260926 02:07:26.644568 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.643941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40175 (local address 127.0.0.1:43246)
0926 02:07:26.644236 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:26.644387 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260926 02:07:26.644850 23715 catalog_manager.cc:4584] TS d6270cf503824b7eaca6b3a00a95a4ed (127.18.159.129:40175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b813ee5b501741eb991a5353445ff3e0: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40175: connect: Connection refused (error 111)
I20260926 02:07:26.645881 24191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.643615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48428 (local address 127.18.159.130:38841)
0926 02:07:26.643850 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:26.643865 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:26.643889 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:07:26.643982 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:26.643987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:26.644096 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:26.644258 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:26.645231 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:26.645237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:26.645241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:07:26.645262 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:26.645300 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:07:26.645313 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:26.645445 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:26.645550 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 02:07:26.645611 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:07:26.646597 24043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:26.698746 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.698234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40175 (local address 127.0.0.1:43262)
0926 02:07:26.698473 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:26.698581 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260926 02:07:26.699937 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.699541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40175 (local address 127.0.0.1:43264)
0926 02:07:26.699711 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:26.699808 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 02:07:26.760680 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.760073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40175 (local address 127.0.0.1:43270)
0926 02:07:26.760341 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:26.760480 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 02:07:26.761940 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.761491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40175 (local address 127.0.0.1:43274)
0926 02:07:26.761655 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:26.761745 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 02:07:26.782068 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 23921
I20260926 02:07:26.798919 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 24051
I20260926 02:07:26.802276 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.799758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.18.159.131:33109)
0926 02:07:26.799936 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:26.799941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:26.799961 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:07:26.800253 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:26.800257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:26.800307 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:26.800424 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:26.801565 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:26.801569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:26.801572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:07:26.801586 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:26.801613 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:07:26.801624 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:26.801741 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:26.801854 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 02:07:26.801915 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:07:26.802933 24173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:26.842964 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.842381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40175 (local address 127.0.0.1:43280)
0926 02:07:26.842645 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:26.842764 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260926 02:07:26.844259 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:26.843752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40175 (local address 127.0.0.1:43294)
0926 02:07:26.843983 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:26.844082 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260926 02:07:26.921802 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 24051
I20260926 02:07:26.941152 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 23700
I20260926 02:07:26.942495 23761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:27.054077 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:27.053290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40175 (local address 127.0.0.1:43296)
0926 02:07:27.053546 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:27.053691 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 02:07:27.054598 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 23700
2026-09-26T02:07:27Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2769 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 02:07:27.100611 19070 mini_oidc.cc:152] Starting JWKS server
I20260926 02:07:27.103312 19070 webserver.cc:460] Webserver started at https://127.0.0.1:35069/ using document root <none> and password file <none>
I20260926 02:07:27.103605 19070 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 02:07:27.104332 19070 webserver.cc:460] Webserver started at http://127.0.0.1:45293/ using document root <none> and password file <none>
2026-09-26T02:07:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:07:27Z Disabled control of system clock
I20260926 02:07:27.136013 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:38549
--webserver_interface=127.18.159.190
--webserver_port=0
--builtin_ntp_servers=127.18.159.148:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:38549
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35069/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388402166281-19070-0/testchainca.cert with env {}
W20260926 02:07:27.530193 24232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:27.530529 24232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:27.530620 24232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:27.538725 24232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:07:27.538913 24232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:27.538964 24232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:27.539026 24232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:07:27.539072 24232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 02:07:27.539224 24232 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 02:07:27.539296 24232 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35069/jwks.json
I20260926 02:07:27.550860 24232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:38549
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:38549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35069/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388402166281-19070-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:27.552305 24232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:27.553745 24232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:27.567214 24238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:27.567274 24240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:27.567305 24237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:27.568302 24232 server_base.cc:1034] running on GCE node
I20260926 02:07:27.569196 24232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:27.570818 24232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:27.572098 24232 hybrid_clock.cc:648] HybridClock initialized: now 1790388447572021 us; error 69 us; skew 500 ppm
I20260926 02:07:27.572660 24232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:07:27.575500 24232 webserver.cc:460] Webserver started at http://127.18.159.190:41951/ using document root <none> and password file <none>
I20260926 02:07:27.576470 24232 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:27.576624 24232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:27.576982 24232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:27.580303 24232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "9bbd6e6a95704849acc2b6caca653d58"
format_stamp: "Formatted at 2026-09-26 02:07:27 on dist-test-slave-03rn"
server_key: "0134e14479a883594aead2da5dbd1f17"
server_key_iv: "7191a9abf758fa7f0b0eb5000b21eb9a"
server_key_version: "encryptionkey@0"
I20260926 02:07:27.581243 24232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "9bbd6e6a95704849acc2b6caca653d58"
format_stamp: "Formatted at 2026-09-26 02:07:27 on dist-test-slave-03rn"
server_key: "0134e14479a883594aead2da5dbd1f17"
server_key_iv: "7191a9abf758fa7f0b0eb5000b21eb9a"
server_key_version: "encryptionkey@0"
I20260926 02:07:27.587245 24232 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260926 02:07:27.591920 24246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:27.593680 24232 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260926 02:07:27.593983 24232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "9bbd6e6a95704849acc2b6caca653d58"
format_stamp: "Formatted at 2026-09-26 02:07:27 on dist-test-slave-03rn"
server_key: "0134e14479a883594aead2da5dbd1f17"
server_key_iv: "7191a9abf758fa7f0b0eb5000b21eb9a"
server_key_version: "encryptionkey@0"
I20260926 02:07:27.594273 24232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:27.622570 24232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:27.623564 24232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:27.623921 24232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:27.643973 24232 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:38549
I20260926 02:07:27.643990 24297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:38549 every 8 connection(s)
I20260926 02:07:27.645512 24232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:07:27.649927 24298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:27.652051 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 24232
I20260926 02:07:27.652321 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:07:27.653116 19070 external_mini_cluster.cc:1428] Setting key 2b1ecb6e5382a97360c0f8f07797353d
I20260926 02:07:27.663358 24298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58: Bootstrap starting.
I20260926 02:07:27.667101 24298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:27.668421 24298 log.cc:824] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:27.668576 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:27.655390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57880 (local address 127.18.159.190:38549)
0926 02:07:27.656725 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:27.656741 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:27.656798 (+ 57us) server_negotiation.cc:407] Connection header received
0926 02:07:27.658618 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:27.658647 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:27.659224 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:27.659727 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:27.660432 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:27.661841 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:27.663326 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:27.664230 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:27.664470 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:27.665786 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:27.665794 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:27.665809 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:07:27.666423 (+ 614us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:27.666525 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:07:27.666549 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:27.666704 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:27.666953 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 02:07:27.667393 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792,"thread_start_us":442,"threads_started":1}
I20260926 02:07:27.671741 24298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58: No bootstrap required, opened a new log
I20260926 02:07:27.672994 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:07:27.680907 24298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bbd6e6a95704849acc2b6caca653d58" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38549 } }
I20260926 02:07:27.681320 24298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:27.681445 24298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bbd6e6a95704849acc2b6caca653d58, State: Initialized, Role: FOLLOWER
I20260926 02:07:27.682096 24298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bbd6e6a95704849acc2b6caca653d58" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38549 } }
I20260926 02:07:27.682315 24298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:07:27.682473 24298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:07:27.682672 24298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:27.685093 24298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bbd6e6a95704849acc2b6caca653d58" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38549 } }
I20260926 02:07:27.685745 24298 leader_election.cc:302] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bbd6e6a95704849acc2b6caca653d58; no voters:
I20260926 02:07:27.686563 24298 leader_election.cc:288] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:07:27.686738 24303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:27.687531 24303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58 [term 1 LEADER]: Becoming Leader. State: Replica: 9bbd6e6a95704849acc2b6caca653d58, State: Running, Role: LEADER
I20260926 02:07:27.688128 24303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bbd6e6a95704849acc2b6caca653d58" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38549 } }
I20260926 02:07:27.689216 24298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:07:27.691777 24304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bbd6e6a95704849acc2b6caca653d58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bbd6e6a95704849acc2b6caca653d58" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38549 } } }
I20260926 02:07:27.692535 24304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58 [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:27.691890 24305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bbd6e6a95704849acc2b6caca653d58. Latest consensus state: current_term: 1 leader_uuid: "9bbd6e6a95704849acc2b6caca653d58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bbd6e6a95704849acc2b6caca653d58" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38549 } } }
I20260926 02:07:27.698167 24305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58 [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:27.699731 24310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:07:27.704913 24310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:07:27.714320 24310 catalog_manager.cc:1347] Generated new cluster ID: fed1fe5fb33f42a58dd89e1b7daac2ef
I20260926 02:07:27.714465 24310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:07:27.730477 24310 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:07:27.731936 24310 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:07:27.740228 24310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bbd6e6a95704849acc2b6caca653d58: Generated new TSK 0
I20260926 02:07:27.741187 24310 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 02:07:27.785739 24328 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 02:07:27.785972 24328 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 02:07:27.787155 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:27.763630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.18.159.190:38549)
0926 02:07:27.763911 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:27.763923 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:27.764711 (+ 788us) server_negotiation.cc:407] Connection header received
0926 02:07:27.765034 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:27.765044 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:27.765139 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:27.765306 (+ 167us) server_negotiation.cc:226] Negotiated authn=JWT
0926 02:07:27.765933 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:27.767192 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:27.768801 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:27.769525 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:27.769683 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:27.770256 (+ 573us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 02:07:27.786433 (+ 16177us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 02:07:27.786825 (+ 392us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57896: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":96}
W20260926 02:07:27.787290 24301 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57896: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 02:07:27.788700 24327 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:27.763509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:38549 (local address 127.0.0.1:57896)
0926 02:07:27.764578 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:27.764596 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:27.764772 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:27.765374 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:27.765378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:27.765389 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0926 02:07:27.765763 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:27.765772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:27.767373 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:27.767378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:27.768647 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:27.768660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:27.769708 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:27.769712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:27.769861 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:27.769932 (+ 71us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 02:07:27.786768 (+ 16836us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 02:07:27.787063 (+ 295us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.159.190:38549: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":875,"thread_start_us":671,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.18.159.190:38549: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 02:07:27.790578 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 24232
I20260926 02:07:27.792008 24293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:27.887190 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 24232
2026-09-26T02:07:27Z chronyd exiting
I20260926 02:07:27.938409 19070 test_util.cc:175] -----------------------------------------------
I20260926 02:07:27.938606 19070 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388402166281-19070-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (842 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 02:07:27.942552 19070 mini_oidc.cc:152] Starting JWKS server
I20260926 02:07:27.944522 19070 webserver.cc:460] Webserver started at https://127.0.0.1:40525/ using document root <none> and password file <none>
I20260926 02:07:27.944767 19070 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 02:07:27.945443 19070 webserver.cc:460] Webserver started at http://127.0.0.1:45205/ using document root <none> and password file <none>
2026-09-26T02:07:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:07:27Z Disabled control of system clock
I20260926 02:07:27.974704 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:39067
--webserver_interface=127.18.159.190
--webserver_port=0
--builtin_ntp_servers=127.18.159.148:43035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:39067
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40525/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388402166281-19070-0/testchainca.cert with env {}
W20260926 02:07:28.313989 24335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:28.314376 24335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:28.314544 24335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:28.322682 24335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:07:28.322835 24335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:28.322917 24335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:28.323004 24335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:07:28.323076 24335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 02:07:28.323163 24335 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 02:07:28.323222 24335 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40525/jwks.json
I20260926 02:07:28.333407 24335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:43035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:39067
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:39067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40525/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388402166281-19070-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:28.334853 24335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:28.336300 24335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:28.348785 24340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:28.348860 24341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:28.348838 24343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:28.349699 24335 server_base.cc:1034] running on GCE node
I20260926 02:07:28.350550 24335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:28.352066 24335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:28.353278 24335 hybrid_clock.cc:648] HybridClock initialized: now 1790388448353237 us; error 41 us; skew 500 ppm
I20260926 02:07:28.353669 24335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:07:28.356070 24335 webserver.cc:460] Webserver started at http://127.18.159.190:45073/ using document root <none> and password file <none>
I20260926 02:07:28.356853 24335 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:28.356983 24335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:28.357331 24335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:28.360471 24335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "b277a839df484bfcab4a022ee5de3cef"
format_stamp: "Formatted at 2026-09-26 02:07:28 on dist-test-slave-03rn"
server_key: "189668cf37617f1eef71f437a4c868f2"
server_key_iv: "b8b366dd217be0f901768c1e2587ab8f"
server_key_version: "encryptionkey@0"
I20260926 02:07:28.361320 24335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "b277a839df484bfcab4a022ee5de3cef"
format_stamp: "Formatted at 2026-09-26 02:07:28 on dist-test-slave-03rn"
server_key: "189668cf37617f1eef71f437a4c868f2"
server_key_iv: "b8b366dd217be0f901768c1e2587ab8f"
server_key_version: "encryptionkey@0"
I20260926 02:07:28.367669 24335 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 02:07:28.372367 24349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:28.374284 24335 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:07:28.374537 24335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "b277a839df484bfcab4a022ee5de3cef"
format_stamp: "Formatted at 2026-09-26 02:07:28 on dist-test-slave-03rn"
server_key: "189668cf37617f1eef71f437a4c868f2"
server_key_iv: "b8b366dd217be0f901768c1e2587ab8f"
server_key_version: "encryptionkey@0"
I20260926 02:07:28.374822 24335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:28.410076 24335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:28.411024 24335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:28.411366 24335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:28.430135 24335 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:39067
I20260926 02:07:28.430161 24400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:39067 every 8 connection(s)
I20260926 02:07:28.431612 24335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:07:28.435681 24401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:28.436208 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 24335
I20260926 02:07:28.436398 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:07:28.437083 19070 external_mini_cluster.cc:1428] Setting key 32bc42e51d4b5534c55bde1d8ee242d8
I20260926 02:07:28.449318 24401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef: Bootstrap starting.
I20260926 02:07:28.451700 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:28.439030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39886 (local address 127.18.159.190:39067)
0926 02:07:28.440037 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:28.440048 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:28.440090 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:07:28.442093 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:28.442121 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:28.442639 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:28.443127 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:28.443730 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:28.445078 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:28.446936 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:28.447667 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:28.447889 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:28.449128 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:28.449136 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:28.449152 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:07:28.449771 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:28.449866 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:07:28.449897 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:28.450028 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:28.450266 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 02:07:28.450604 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":337,"threads_started":1}
I20260926 02:07:28.453131 24401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:28.454650 24401 log.cc:824] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:28.456557 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:07:28.458606 24401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef: No bootstrap required, opened a new log
I20260926 02:07:28.466995 24401 raft_consensus.cc:348] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b277a839df484bfcab4a022ee5de3cef" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39067 } }
I20260926 02:07:28.467379 24401 raft_consensus.cc:374] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:28.467448 24401 raft_consensus.cc:729] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b277a839df484bfcab4a022ee5de3cef, State: Initialized, Role: FOLLOWER
I20260926 02:07:28.467979 24401 consensus_queue.cc:260] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b277a839df484bfcab4a022ee5de3cef" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39067 } }
I20260926 02:07:28.468245 24401 raft_consensus.cc:388] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:07:28.468365 24401 raft_consensus.cc:482] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:07:28.468495 24401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:28.470672 24401 raft_consensus.cc:504] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b277a839df484bfcab4a022ee5de3cef" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39067 } }
I20260926 02:07:28.471170 24401 leader_election.cc:302] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b277a839df484bfcab4a022ee5de3cef; no voters:
I20260926 02:07:28.471722 24401 leader_election.cc:288] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:07:28.471884 24406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:28.472708 24406 raft_consensus.cc:686] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef [term 1 LEADER]: Becoming Leader. State: Replica: b277a839df484bfcab4a022ee5de3cef, State: Running, Role: LEADER
I20260926 02:07:28.473320 24406 consensus_queue.cc:237] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b277a839df484bfcab4a022ee5de3cef" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39067 } }
I20260926 02:07:28.474004 24401 sys_catalog.cc:564] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:07:28.476369 24408 sys_catalog.cc:455] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef [sys.catalog]: SysCatalogTable state changed. Reason: New leader b277a839df484bfcab4a022ee5de3cef. Latest consensus state: current_term: 1 leader_uuid: "b277a839df484bfcab4a022ee5de3cef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b277a839df484bfcab4a022ee5de3cef" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39067 } } }
I20260926 02:07:28.476414 24407 sys_catalog.cc:455] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b277a839df484bfcab4a022ee5de3cef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b277a839df484bfcab4a022ee5de3cef" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39067 } } }
I20260926 02:07:28.476878 24407 sys_catalog.cc:458] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:28.476878 24408 sys_catalog.cc:458] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:28.477819 24413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:07:28.484375 24413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:07:28.495187 24413 catalog_manager.cc:1347] Generated new cluster ID: d868e71d6021433eb22993db02f4970d
I20260926 02:07:28.495322 24413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:07:28.507895 24413 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:07:28.509886 24413 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:07:28.531647 24413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b277a839df484bfcab4a022ee5de3cef: Generated new TSK 0
I20260926 02:07:28.532967 24413 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 02:07:28.566485 24431 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 02:07:28.566638 24431 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 02:07:28.567884 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:28.547188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39894 (local address 127.18.159.190:39067)
0926 02:07:28.547399 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:28.547405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:28.547836 (+ 431us) server_negotiation.cc:407] Connection header received
0926 02:07:28.548075 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:28.548082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:28.548220 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:28.548411 (+ 191us) server_negotiation.cc:226] Negotiated authn=JWT
0926 02:07:28.548864 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:28.550038 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:28.551494 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:28.552261 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:28.552397 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:28.553038 (+ 641us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 02:07:28.567149 (+ 14111us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 02:07:28.567570 (+ 421us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39894: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 02:07:28.568019 24404 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39894: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 02:07:28.568064 24430 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:28.546992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:39067 (local address 127.0.0.1:39894)
0926 02:07:28.547725 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:28.547741 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:28.547873 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:28.548423 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:28.548428 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:28.548440 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0926 02:07:28.548688 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:28.548695 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:28.550221 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:28.550225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:28.551333 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:28.551346 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:28.552525 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:28.552532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:28.552685 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:28.552724 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 02:07:28.567438 (+ 14714us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 02:07:28.567677 (+ 239us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.159.190:39067: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":427,"threads_started":1}
I20260926 02:07:28.569443 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 24335
I20260926 02:07:28.570839 24396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:28.664512 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 24335
2026-09-26T02:07:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (777 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 02:07:28.720472 19070 mini_oidc.cc:152] Starting JWKS server
I20260926 02:07:28.722412 19070 webserver.cc:460] Webserver started at https://127.0.0.1:36715/ using document root <none> and password file <none>
I20260926 02:07:28.722648 19070 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 02:07:28.723344 19070 webserver.cc:460] Webserver started at http://127.0.0.1:43925/ using document root <none> and password file <none>
2026-09-26T02:07:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:07:28Z Disabled control of system clock
I20260926 02:07:28.753206 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:38381
--webserver_interface=127.18.159.190
--webserver_port=0
--builtin_ntp_servers=127.18.159.148:40213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:38381
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36715/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 02:07:29.091447 24438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:29.091852 24438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:29.092020 24438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:29.099840 24438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:07:29.100030 24438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:29.100131 24438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:29.100281 24438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:07:29.100358 24438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 02:07:29.100455 24438 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 02:07:29.100535 24438 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36715/jwks.json
I20260926 02:07:29.110061 24438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:40213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:38381
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:38381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36715/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:29.111466 24438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:29.113315 24438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:29.125758 24443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:29.125762 24446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:29.125765 24444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:29.126618 24438 server_base.cc:1034] running on GCE node
I20260926 02:07:29.127465 24438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:29.129045 24438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:29.130323 24438 hybrid_clock.cc:648] HybridClock initialized: now 1790388449130242 us; error 61 us; skew 500 ppm
I20260926 02:07:29.130801 24438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:07:29.133646 24438 webserver.cc:460] Webserver started at http://127.18.159.190:42679/ using document root <none> and password file <none>
I20260926 02:07:29.134495 24438 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:29.134598 24438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:29.135115 24438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:29.138583 24438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "5ba8d0964f404ca0873144c1405e08ad"
format_stamp: "Formatted at 2026-09-26 02:07:29 on dist-test-slave-03rn"
server_key: "175725baf2eeb5ef2c42e7d95e7ec336"
server_key_iv: "1600c5fdf76da90f905cf754b0540c33"
server_key_version: "encryptionkey@0"
I20260926 02:07:29.139537 24438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "5ba8d0964f404ca0873144c1405e08ad"
format_stamp: "Formatted at 2026-09-26 02:07:29 on dist-test-slave-03rn"
server_key: "175725baf2eeb5ef2c42e7d95e7ec336"
server_key_iv: "1600c5fdf76da90f905cf754b0540c33"
server_key_version: "encryptionkey@0"
I20260926 02:07:29.145416 24438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 02:07:29.149477 24452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:29.151253 24438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:07:29.151473 24438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "5ba8d0964f404ca0873144c1405e08ad"
format_stamp: "Formatted at 2026-09-26 02:07:29 on dist-test-slave-03rn"
server_key: "175725baf2eeb5ef2c42e7d95e7ec336"
server_key_iv: "1600c5fdf76da90f905cf754b0540c33"
server_key_version: "encryptionkey@0"
I20260926 02:07:29.151738 24438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:29.176816 24438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:29.177840 24438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:29.178227 24438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:29.199695 24438 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:38381
I20260926 02:07:29.199729 24503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:38381 every 8 connection(s)
I20260926 02:07:29.201298 24438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:07:29.204864 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 24438
I20260926 02:07:29.205176 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:07:29.205806 19070 external_mini_cluster.cc:1428] Setting key 3d7d0f90d8c49fc50668cdf37454e91c
I20260926 02:07:29.206043 24504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:29.224018 24504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad: Bootstrap starting.
I20260926 02:07:29.224496 24507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:29.207965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44606 (local address 127.18.159.190:38381)
0926 02:07:29.209241 (+ 1276us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:29.209260 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:29.209328 (+ 68us) server_negotiation.cc:407] Connection header received
0926 02:07:29.211801 (+ 2473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:29.211843 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:29.212732 (+ 889us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:29.213357 (+ 625us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:29.214091 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:29.216506 (+ 2415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:29.218057 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:29.219345 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:29.219588 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:29.221113 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:29.221131 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:29.221156 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:07:29.222024 (+ 868us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:29.222138 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:07:29.222174 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:29.222304 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:29.222587 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 02:07:29.222965 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":447,"threads_started":1}
I20260926 02:07:29.227839 24504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:29.228931 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:07:29.229250 24504 log.cc:824] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:29.232837 24504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad: No bootstrap required, opened a new log
I20260926 02:07:29.241467 24504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba8d0964f404ca0873144c1405e08ad" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38381 } }
I20260926 02:07:29.241922 24504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:29.242080 24504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba8d0964f404ca0873144c1405e08ad, State: Initialized, Role: FOLLOWER
I20260926 02:07:29.242878 24504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba8d0964f404ca0873144c1405e08ad" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38381 } }
I20260926 02:07:29.243192 24504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:07:29.243372 24504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:07:29.243592 24504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:29.246133 24504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba8d0964f404ca0873144c1405e08ad" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38381 } }
I20260926 02:07:29.246752 24504 leader_election.cc:302] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ba8d0964f404ca0873144c1405e08ad; no voters:
I20260926 02:07:29.247449 24504 leader_election.cc:288] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:07:29.247635 24509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:29.249086 24509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad [term 1 LEADER]: Becoming Leader. State: Replica: 5ba8d0964f404ca0873144c1405e08ad, State: Running, Role: LEADER
I20260926 02:07:29.249784 24509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba8d0964f404ca0873144c1405e08ad" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38381 } }
I20260926 02:07:29.250130 24504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:07:29.252972 24511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ba8d0964f404ca0873144c1405e08ad. Latest consensus state: current_term: 1 leader_uuid: "5ba8d0964f404ca0873144c1405e08ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba8d0964f404ca0873144c1405e08ad" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38381 } } }
I20260926 02:07:29.253525 24511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:29.254334 24510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ba8d0964f404ca0873144c1405e08ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba8d0964f404ca0873144c1405e08ad" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38381 } } }
I20260926 02:07:29.254772 24510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:29.256251 24518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:07:29.262501 24518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:07:29.273125 24518 catalog_manager.cc:1347] Generated new cluster ID: 65b77f15d10a405fb17cd92b498e66fe
I20260926 02:07:29.273263 24518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:07:29.290050 24518 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:07:29.291688 24518 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:07:29.306205 24518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ba8d0964f404ca0873144c1405e08ad: Generated new TSK 0
I20260926 02:07:29.307276 24518 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:07:29.344448 24507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:29.335278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44610 (local address 127.18.159.190:38381)
0926 02:07:29.335502 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:29.335509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:29.336101 (+ 592us) server_negotiation.cc:407] Connection header received
0926 02:07:29.336403 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:29.336416 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:29.336527 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:29.336732 (+ 205us) server_negotiation.cc:226] Negotiated authn=JWT
0926 02:07:29.337213 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:29.338344 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:29.339786 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:29.340550 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:29.340693 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:29.341297 (+ 604us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 02:07:29.343720 (+ 2423us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36715/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 02:07:29.344086 (+ 366us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44610: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36715/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 02:07:29.344592 24507 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44610: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36715/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 02:07:29.346233 24533 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:29.335190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:38381 (local address 127.0.0.1:44610)
0926 02:07:29.335971 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:29.335990 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:29.336134 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:29.336756 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:29.336759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:29.336769 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0926 02:07:29.337051 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:29.337059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:29.338544 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:29.338548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:29.339634 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:29.339644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:29.340772 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:29.340777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:29.340935 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:29.340971 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 02:07:29.343973 (+ 3002us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36715/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 02:07:29.345793 (+ 1820us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.159.190:38381: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36715/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":1458,"thread_start_us":466,"threads_started":1}
I20260926 02:07:29.348582 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 24438
I20260926 02:07:29.349898 24499 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 02:07:29.356737 24534 webserver.cc:533] Webserver: sslize failed:
E20260926 02:07:29.356873 24534 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 02:07:29.446448 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 24438
2026-09-26T02:07:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (782 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T02:07:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:07:29Z Disabled control of system clock
I20260926 02:07:29.532022 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:44857
--webserver_interface=127.18.159.190
--webserver_port=0
--builtin_ntp_servers=127.18.159.148:33269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:44857
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388402166281-19070-0/testchainca.cert with env {}
W20260926 02:07:29.886207 24539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:29.886561 24539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:29.886745 24539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:29.895241 24539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:07:29.895385 24539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:29.895422 24539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:29.895555 24539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:07:29.895641 24539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 02:07:29.895735 24539 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 02:07:29.895790 24539 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 02:07:29.906451 24539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:33269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:44857
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:44857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388402166281-19070-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:29.907902 24539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:29.909307 24539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:29.921833 24544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:29.921855 24547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:29.921765 24545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:29.923112 24539 server_base.cc:1034] running on GCE node
I20260926 02:07:29.923951 24539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:29.925563 24539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:29.926868 24539 hybrid_clock.cc:648] HybridClock initialized: now 1790388449926800 us; error 64 us; skew 500 ppm
I20260926 02:07:29.927275 24539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:07:29.934445 24539 webserver.cc:460] Webserver started at http://127.18.159.190:35895/ using document root <none> and password file <none>
I20260926 02:07:29.935230 24539 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:29.935341 24539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:29.935607 24539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:29.938652 24539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "2e67a778503f4c7bb96ebfcc02e81d88"
format_stamp: "Formatted at 2026-09-26 02:07:29 on dist-test-slave-03rn"
server_key: "2dbb75911e9d3ea3f71ca2ee22d94cfa"
server_key_iv: "6eaa60d51bb242baa3e9200f5c2a15f5"
server_key_version: "encryptionkey@0"
I20260926 02:07:29.939473 24539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "2e67a778503f4c7bb96ebfcc02e81d88"
format_stamp: "Formatted at 2026-09-26 02:07:29 on dist-test-slave-03rn"
server_key: "2dbb75911e9d3ea3f71ca2ee22d94cfa"
server_key_iv: "6eaa60d51bb242baa3e9200f5c2a15f5"
server_key_version: "encryptionkey@0"
I20260926 02:07:29.945021 24539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 02:07:29.948952 24553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:29.950531 24539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:07:29.950763 24539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "2e67a778503f4c7bb96ebfcc02e81d88"
format_stamp: "Formatted at 2026-09-26 02:07:29 on dist-test-slave-03rn"
server_key: "2dbb75911e9d3ea3f71ca2ee22d94cfa"
server_key_iv: "6eaa60d51bb242baa3e9200f5c2a15f5"
server_key_version: "encryptionkey@0"
I20260926 02:07:29.951030 24539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:29.990017 24539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:29.990945 24539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:29.991262 24539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:30.011197 24539 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:44857
I20260926 02:07:30.011216 24604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:44857 every 8 connection(s)
I20260926 02:07:30.012869 24539 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:07:30.014865 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 24539
I20260926 02:07:30.015076 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:07:30.015897 19070 external_mini_cluster.cc:1428] Setting key 07915fbb34b71489dd3688c408f366d0
I20260926 02:07:30.018306 24605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:30.034581 24608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:30.017979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54362 (local address 127.18.159.190:44857)
0926 02:07:30.019025 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:30.019036 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:30.019080 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:07:30.021281 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:30.021312 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:30.021963 (+ 651us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:30.022442 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:30.023698 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:30.025865 (+ 2167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:30.027850 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:30.028978 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:30.029166 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:30.030806 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:30.030818 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:30.030844 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:07:30.032047 (+ 1203us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:30.032213 (+ 166us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:07:30.032254 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:30.032385 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:30.032710 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 02:07:30.033172 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":385,"threads_started":1}
I20260926 02:07:30.034996 24605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88: Bootstrap starting.
I20260926 02:07:30.039108 24605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:30.039395 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:07:30.040336 24605 log.cc:824] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:30.043982 24605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88: No bootstrap required, opened a new log
I20260926 02:07:30.052268 24605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e67a778503f4c7bb96ebfcc02e81d88" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44857 } }
I20260926 02:07:30.052883 24605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:30.053045 24605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e67a778503f4c7bb96ebfcc02e81d88, State: Initialized, Role: FOLLOWER
I20260926 02:07:30.053828 24605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e67a778503f4c7bb96ebfcc02e81d88" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44857 } }
I20260926 02:07:30.054105 24605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:07:30.054246 24605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:07:30.054433 24605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:30.056993 24605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e67a778503f4c7bb96ebfcc02e81d88" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44857 } }
I20260926 02:07:30.057595 24605 leader_election.cc:302] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e67a778503f4c7bb96ebfcc02e81d88; no voters:
I20260926 02:07:30.058230 24605 leader_election.cc:288] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:07:30.058442 24610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:30.059227 24610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88 [term 1 LEADER]: Becoming Leader. State: Replica: 2e67a778503f4c7bb96ebfcc02e81d88, State: Running, Role: LEADER
I20260926 02:07:30.059791 24610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e67a778503f4c7bb96ebfcc02e81d88" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44857 } }
I20260926 02:07:30.060580 24605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:07:30.062601 24612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e67a778503f4c7bb96ebfcc02e81d88. Latest consensus state: current_term: 1 leader_uuid: "2e67a778503f4c7bb96ebfcc02e81d88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e67a778503f4c7bb96ebfcc02e81d88" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44857 } } }
I20260926 02:07:30.063052 24612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88 [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:30.063865 24611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e67a778503f4c7bb96ebfcc02e81d88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e67a778503f4c7bb96ebfcc02e81d88" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44857 } } }
I20260926 02:07:30.064217 24616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:07:30.064278 24611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88 [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:30.071111 24616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:07:30.081754 24616 catalog_manager.cc:1347] Generated new cluster ID: f32c61e0159843d5a18252a37f9ba7c3
I20260926 02:07:30.081983 24616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:07:30.113333 24616 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:07:30.114785 24616 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:07:30.123646 24616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e67a778503f4c7bb96ebfcc02e81d88: Generated new TSK 0
I20260926 02:07:30.124686 24616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:07:30.198081 24608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:30.147847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54378 (local address 127.18.159.190:44857)
0926 02:07:30.148062 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:30.148069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:30.148639 (+ 570us) server_negotiation.cc:407] Connection header received
0926 02:07:30.148895 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:30.148901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:30.149001 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:30.149182 (+ 181us) server_negotiation.cc:226] Negotiated authn=JWT
0926 02:07:30.149698 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:30.150878 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:30.152545 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:30.153314 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:30.153473 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:30.154105 (+ 632us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 02:07:30.197319 (+ 43214us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 02:07:30.197766 (+ 447us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54378: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 02:07:30.198221 24608 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54378: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 02:07:30.198285 24634 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:30.147784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44857 (local address 127.0.0.1:54378)
0926 02:07:30.148520 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:30.148537 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:30.148672 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:30.149257 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:30.149262 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:30.149275 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0926 02:07:30.149523 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:30.149531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:30.151065 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:30.151069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:30.152369 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:30.152385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:30.153560 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:30.153565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:30.153717 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:30.153764 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 02:07:30.197660 (+ 43896us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 02:07:30.197916 (+ 256us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.159.190:44857: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":408,"threads_started":1}
I20260926 02:07:30.199617 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 24539
I20260926 02:07:30.201036 24600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:30.301614 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 24539
2026-09-26T02:07:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (833 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (444 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (431 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:07:31 dist-test-slave-03rn krb5kdc[24649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:07:31 dist-test-slave-03rn krb5kdc[24649](info): set up 2 sockets
Sep 26 02:07:31 dist-test-slave-03rn krb5kdc[24649](info): commencing operation
krb5kdc: starting...
W20260926 02:07:32.277350 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:07:32 dist-test-slave-03rn krb5kdc[24649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:07:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:07:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:07:32.501091 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:45383
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:45383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:32.850347 24662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:32.850677 24662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:32.850775 24662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:32.858820 24662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:07:32.858955 24662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:32.859002 24662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:32.859052 24662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:07:32.859119 24662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:07:32.869151 24662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:45383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:45383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:32.870460 24662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:32.871873 24662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:32.884007 24667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:32.883953 24668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:32.884259 24670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:32.884864 24662 server_base.cc:1034] running on GCE node
I20260926 02:07:32.885622 24662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:32.887050 24662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:32.888275 24662 hybrid_clock.cc:648] HybridClock initialized: now 1790388452888190 us; error 66 us; skew 500 ppm
I20260926 02:07:32.888597 24662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:32 dist-test-slave-03rn krb5kdc[24649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388452, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:32.891764 24662 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:07:32.893414 24662 webserver.cc:460] Webserver started at http://127.18.159.190:41685/ using document root <none> and password file <none>
I20260926 02:07:32.894168 24662 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:32.894313 24662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:32.894659 24662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:32.898037 24662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "9850a6c3706e46678a2e295a616fbc7d"
format_stamp: "Formatted at 2026-09-26 02:07:32 on dist-test-slave-03rn"
server_key: "e3d1e0a6dbe3874eba14ed12dceeeb83"
server_key_iv: "c9ff43c84d48d2ec21110696b04a5da8"
server_key_version: "encryptionkey@0"
I20260926 02:07:32.898979 24662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "9850a6c3706e46678a2e295a616fbc7d"
format_stamp: "Formatted at 2026-09-26 02:07:32 on dist-test-slave-03rn"
server_key: "e3d1e0a6dbe3874eba14ed12dceeeb83"
server_key_iv: "c9ff43c84d48d2ec21110696b04a5da8"
server_key_version: "encryptionkey@0"
I20260926 02:07:32.904798 24662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 02:07:32.908499 24677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:32.910013 24662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:07:32.910233 24662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "9850a6c3706e46678a2e295a616fbc7d"
format_stamp: "Formatted at 2026-09-26 02:07:32 on dist-test-slave-03rn"
server_key: "e3d1e0a6dbe3874eba14ed12dceeeb83"
server_key_iv: "c9ff43c84d48d2ec21110696b04a5da8"
server_key_version: "encryptionkey@0"
I20260926 02:07:32.910471 24662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:32.951027 24662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:32.955641 24662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:32.955976 24662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:32.975107 24662 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:45383
I20260926 02:07:32.975140 24728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:45383 every 8 connection(s)
I20260926 02:07:32.976747 24662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:07:32.981073 24729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:32.983322 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 24662
I20260926 02:07:32.983569 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:07:32.984376 19070 external_mini_cluster.cc:1428] Setting key c9fbca8cf1c9ad64903ec738f6c4c1a9
I20260926 02:07:32.995901 24729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d: Bootstrap starting.
I20260926 02:07:33.000042 24729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:33.001256 24729 log.cc:824] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:07:32 dist-test-slave-03rn krb5kdc[24649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:33.004951 24729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d: No bootstrap required, opened a new log
I20260926 02:07:33.011888 24732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:32.987114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.18.159.190:45383)
0926 02:07:32.988259 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:32.988277 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:32.988761 (+ 484us) server_negotiation.cc:407] Connection header received
0926 02:07:32.991253 (+ 2492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:32.991295 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:32.991935 (+ 640us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:32.992504 (+ 569us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:32.993857 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:32.995978 (+ 2121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:32.997527 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:32.998733 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:32.998982 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:33.003757 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:33.003767 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:33.003784 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:33.003823 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:33.007211 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:33.008283 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:33.008291 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:33.008302 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:33.008519 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:33.008965 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:33.008969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:33.008971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:33.009540 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:33.009754 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:33.010291 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 02:07:33.010761 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":825,"thread_start_us":502,"threads_started":1}
I20260926 02:07:33.014468 24729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9850a6c3706e46678a2e295a616fbc7d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45383 } }
I20260926 02:07:33.014957 24729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:33.015131 24729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9850a6c3706e46678a2e295a616fbc7d, State: Initialized, Role: FOLLOWER
I20260926 02:07:33.015935 24729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9850a6c3706e46678a2e295a616fbc7d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45383 } }
I20260926 02:07:33.016245 24729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:07:33.016417 24729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:07:33.016621 24729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:33.018962 24729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9850a6c3706e46678a2e295a616fbc7d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45383 } }
I20260926 02:07:33.019582 24729 leader_election.cc:302] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9850a6c3706e46678a2e295a616fbc7d; no voters:
I20260926 02:07:33.020324 24729 leader_election.cc:288] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:07:33.020457 24734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:33.021167 24734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d [term 1 LEADER]: Becoming Leader. State: Replica: 9850a6c3706e46678a2e295a616fbc7d, State: Running, Role: LEADER
I20260926 02:07:33.021675 24734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9850a6c3706e46678a2e295a616fbc7d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45383 } }
I20260926 02:07:33.022504 24729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:07:33.024869 24736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9850a6c3706e46678a2e295a616fbc7d. Latest consensus state: current_term: 1 leader_uuid: "9850a6c3706e46678a2e295a616fbc7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9850a6c3706e46678a2e295a616fbc7d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45383 } } }
I20260926 02:07:33.026238 24736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:33.026042 24735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9850a6c3706e46678a2e295a616fbc7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9850a6c3706e46678a2e295a616fbc7d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45383 } } }
I20260926 02:07:33.027081 24735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:33.037706 24743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:07:33.043370 24743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:07:33.052373 24743 catalog_manager.cc:1347] Generated new cluster ID: b619f538f61047788bfc20e32f7d4d92
I20260926 02:07:33.052480 24743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:07:33.070366 24743 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:07:33.071784 24743 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:07:33.079682 24743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9850a6c3706e46678a2e295a616fbc7d: Generated new TSK 0
I20260926 02:07:33.080914 24743 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.159.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
I20260926 02:07:33.152148 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:45383
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:33.510231 24755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:33.510578 24755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:33.510756 24755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:33.518572 24755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:33.518747 24755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:33.518945 24755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:07:33.529299 24755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:45383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:33.530668 24755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:33.532073 24755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:33.546156 24760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:33.546182 24763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:33.546113 24761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:33.547072 24755 server_base.cc:1034] running on GCE node
I20260926 02:07:33.547684 24755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:33.548727 24755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:33.549940 24755 hybrid_clock.cc:648] HybridClock initialized: now 1790388453549881 us; error 47 us; skew 500 ppm
I20260926 02:07:33.550314 24755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:33 dist-test-slave-03rn krb5kdc[24649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388453, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:33.553714 24755 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:07:33.555160 24755 webserver.cc:460] Webserver started at http://127.18.159.129:34693/ using document root <none> and password file <none>
I20260926 02:07:33.555842 24755 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:33.555945 24755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:33.556300 24755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:33.559348 24755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "c8e5112dfbd9435a823c233203ede767"
format_stamp: "Formatted at 2026-09-26 02:07:33 on dist-test-slave-03rn"
server_key: "bcbb1311244d6dda50d86a7be6c31ae8"
server_key_iv: "67ec6abc16d8293b5c2e8e17bb4fe815"
server_key_version: "encryptionkey@0"
I20260926 02:07:33.560149 24755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "c8e5112dfbd9435a823c233203ede767"
format_stamp: "Formatted at 2026-09-26 02:07:33 on dist-test-slave-03rn"
server_key: "bcbb1311244d6dda50d86a7be6c31ae8"
server_key_iv: "67ec6abc16d8293b5c2e8e17bb4fe815"
server_key_version: "encryptionkey@0"
I20260926 02:07:33.565663 24755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:07:33.569190 24770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:33.570632 24755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:07:33.570823 24755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "c8e5112dfbd9435a823c233203ede767"
format_stamp: "Formatted at 2026-09-26 02:07:33 on dist-test-slave-03rn"
server_key: "bcbb1311244d6dda50d86a7be6c31ae8"
server_key_iv: "67ec6abc16d8293b5c2e8e17bb4fe815"
server_key_version: "encryptionkey@0"
I20260926 02:07:33.571044 24755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:33.596740 24755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:33.601387 24755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:33.601701 24755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:33.602711 24755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:33.604102 24755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:33.604228 24755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 02:07:33.604375 24755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:33.604447 24755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:33.638296 24755 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:42253
I20260926 02:07:33.638386 24882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:42253 every 8 connection(s)
I20260926 02:07:33.639649 24755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:07:33.647460 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 24755
I20260926 02:07:33.647682 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:07:33.648388 19070 external_mini_cluster.cc:1428] Setting key 9691393b0e6747f07af24051cce930c2
Sep 26 02:07:33 dist-test-slave-03rn krb5kdc[24649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388453, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:33.662681 24885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:33.642417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:45215 (local address 127.18.159.190:45383)
0926 02:07:33.642913 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:33.642921 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:33.644202 (+ 1281us) server_negotiation.cc:407] Connection header received
0926 02:07:33.646043 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:33.646050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:33.646144 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:33.646270 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:33.647897 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:33.648737 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:33.650837 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:33.651250 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:33.651351 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:33.655911 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:33.655923 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:33.655931 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:33.655983 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:33.658975 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:33.660122 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:33.660126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:33.660128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:33.660321 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:33.660848 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:33.660851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:33.660853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:33.661122 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:33.661231 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:33.662217 (+ 986us) server_negotiation.cc:299] Negotiation successful
0926 02:07:33.662405 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":271,"threads_started":1}
I20260926 02:07:33.664378 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:33.643071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:45383 (local address 127.18.159.129:45215)
0926 02:07:33.644021 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:33.644066 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:33.645758 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:33.646509 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:33.646517 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:33.647056 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:33.647663 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:33.647687 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:33.648900 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:33.648906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:33.650690 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:33.650706 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:33.651501 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:33.651511 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:33.651873 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:33.652708 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:33.652744 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:33.655599 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:33.659194 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:33.659209 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:33.659233 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:33.659982 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:33.660444 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:33.660452 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:33.660458 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:33.660736 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:33.661263 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:33.661276 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:33.661763 (+ 487us) client_negotiation.cc:769] Sending connection context
0926 02:07:33.662115 (+ 352us) client_negotiation.cc:240] Negotiation successful
0926 02:07:33.662733 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":371,"threads_started":1}
I20260926 02:07:33.666173 24883 heartbeater.cc:347] Connected to a master server at 127.18.159.190:45383
I20260926 02:07:33.666523 24883 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:33.667347 24883 heartbeater.cc:511] Master 127.18.159.190:45383 requested a full tablet report, sending...
I20260926 02:07:33.669646 24694 ts_manager.cc:194] Registered new tserver with Master: c8e5112dfbd9435a823c233203ede767 (127.18.159.129:42253)
WARNING: no policy specified for kudu/127.18.159.130@KRBTEST.COM; defaulting to no policy
I20260926 02:07:33.671200 24694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:45215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
I20260926 02:07:33.700345 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:45383
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:34.035799 24889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:34.036110 24889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:34.036289 24889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:34.043885 24889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:34.044009 24889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:34.044123 24889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:07:34.054141 24889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:45383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:34.055425 24889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:34.056978 24889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:34.071079 24894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:34.071038 24895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:34.071120 24897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:34.071959 24889 server_base.cc:1034] running on GCE node
I20260926 02:07:34.072697 24889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:34.073711 24889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:34.074971 24889 hybrid_clock.cc:648] HybridClock initialized: now 1790388454074884 us; error 68 us; skew 500 ppm
I20260926 02:07:34.075345 24889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:34 dist-test-slave-03rn krb5kdc[24649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388454, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:34.078677 24889 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:07:34.080340 24889 webserver.cc:460] Webserver started at http://127.18.159.130:45377/ using document root <none> and password file <none>
I20260926 02:07:34.081108 24889 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:34.081259 24889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:34.081578 24889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:34.084694 24889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "7ea3899d8e324965850170634b9039b1"
format_stamp: "Formatted at 2026-09-26 02:07:34 on dist-test-slave-03rn"
server_key: "27d1e4fb1d8e3fed6a68a152d1e8c68a"
server_key_iv: "98efb0fc85c08f8ce465acc13bcb1aed"
server_key_version: "encryptionkey@0"
I20260926 02:07:34.085537 24889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "7ea3899d8e324965850170634b9039b1"
format_stamp: "Formatted at 2026-09-26 02:07:34 on dist-test-slave-03rn"
server_key: "27d1e4fb1d8e3fed6a68a152d1e8c68a"
server_key_iv: "98efb0fc85c08f8ce465acc13bcb1aed"
server_key_version: "encryptionkey@0"
I20260926 02:07:34.091354 24889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 02:07:34.095302 24904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:34.096856 24889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:07:34.097075 24889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "7ea3899d8e324965850170634b9039b1"
format_stamp: "Formatted at 2026-09-26 02:07:34 on dist-test-slave-03rn"
server_key: "27d1e4fb1d8e3fed6a68a152d1e8c68a"
server_key_iv: "98efb0fc85c08f8ce465acc13bcb1aed"
server_key_version: "encryptionkey@0"
I20260926 02:07:34.097319 24889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:34.116425 24889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:34.120988 24889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:34.121351 24889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:34.122380 24889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:34.123822 24889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:34.123925 24889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:34.124086 24889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:34.124161 24889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:34.158876 24889 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:38413
I20260926 02:07:34.158900 25016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:38413 every 8 connection(s)
I20260926 02:07:34.160328 24889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:07:34.162485 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 24889
I20260926 02:07:34.162662 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:07:34.163228 19070 external_mini_cluster.cc:1428] Setting key 0dfbced137a415c740428b78fbc2eca0
Sep 26 02:07:34 dist-test-slave-03rn krb5kdc[24649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388454, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:34.183902 25019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:34.163619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:56159 (local address 127.18.159.190:45383)
0926 02:07:34.164057 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:34.164064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:34.164766 (+ 702us) server_negotiation.cc:407] Connection header received
0926 02:07:34.166760 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:34.166766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:34.166867 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:34.166986 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:34.168662 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:34.169355 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:34.171523 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:34.172369 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:34.172521 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:34.176272 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:34.176282 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:34.176290 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:34.176335 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:34.180300 (+ 3965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:34.181195 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:34.181204 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:34.181210 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:34.181397 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:34.181861 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:34.181869 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:34.181873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:34.182246 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:34.182408 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:34.183133 (+ 725us) server_negotiation.cc:299] Negotiation successful
0926 02:07:34.183404 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":226,"threads_started":1}
WARNING: no policy specified for kudu/127.18.159.131@KRBTEST.COM; defaulting to no policy
I20260926 02:07:34.184729 25020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:34.163768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:45383 (local address 127.18.159.130:56159)
0926 02:07:34.164612 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:34.164650 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:34.166477 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:34.167249 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:34.167259 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:34.167801 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:34.168489 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:34.168509 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:34.169513 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:34.169526 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:34.171336 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:34.171354 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:34.172584 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:34.172590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:34.172798 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:34.173562 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:34.173590 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:34.175949 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:34.180501 (+ 4552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:34.180519 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:34.180536 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:34.181039 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:34.181549 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:34.181552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:34.181554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:34.181736 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:34.182373 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:34.182380 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:34.182748 (+ 368us) client_negotiation.cc:769] Sending connection context
0926 02:07:34.182996 (+ 248us) client_negotiation.cc:240] Negotiation successful
0926 02:07:34.183415 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":356,"threads_started":1}
I20260926 02:07:34.186244 25017 heartbeater.cc:347] Connected to a master server at 127.18.159.190:45383
I20260926 02:07:34.186528 25017 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.131@KRBTEST.COM" created.
I20260926 02:07:34.187325 25017 heartbeater.cc:511] Master 127.18.159.190:45383 requested a full tablet report, sending...
I20260926 02:07:34.188896 24694 ts_manager.cc:194] Registered new tserver with Master: 7ea3899d8e324965850170634b9039b1 (127.18.159.130:38413)
I20260926 02:07:34.189741 24694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:56159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
I20260926 02:07:34.212432 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:45383
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:34.557600 25023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:34.557961 25023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:34.558138 25023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:34.565470 25023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:34.565644 25023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:34.565833 25023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:07:34.575985 25023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:45383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:34.577400 25023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:34.578828 25023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:34.592928 25029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:34.592900 25031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:34.593019 25028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:34.597818 25023 server_base.cc:1034] running on GCE node
I20260926 02:07:34.598507 25023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:34.599540 25023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:34.600782 25023 hybrid_clock.cc:648] HybridClock initialized: now 1790388454600731 us; error 54 us; skew 500 ppm
I20260926 02:07:34.601228 25023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:34 dist-test-slave-03rn krb5kdc[24649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388454, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:34.604578 25023 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:07:34.606235 25023 webserver.cc:460] Webserver started at http://127.18.159.131:36955/ using document root <none> and password file <none>
I20260926 02:07:34.607121 25023 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:34.607260 25023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:34.607589 25023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:34.610684 25023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "78a4771acb57465890b2c46b3dd7c756"
format_stamp: "Formatted at 2026-09-26 02:07:34 on dist-test-slave-03rn"
server_key: "fef3171382bc478d84a380dd958f8e08"
server_key_iv: "232dd1264fff84364e5b05fbba64a94b"
server_key_version: "encryptionkey@0"
I20260926 02:07:34.611528 25023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "78a4771acb57465890b2c46b3dd7c756"
format_stamp: "Formatted at 2026-09-26 02:07:34 on dist-test-slave-03rn"
server_key: "fef3171382bc478d84a380dd958f8e08"
server_key_iv: "232dd1264fff84364e5b05fbba64a94b"
server_key_version: "encryptionkey@0"
I20260926 02:07:34.617087 25023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:07:34.620921 25038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:34.622439 25023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:07:34.622700 25023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "78a4771acb57465890b2c46b3dd7c756"
format_stamp: "Formatted at 2026-09-26 02:07:34 on dist-test-slave-03rn"
server_key: "fef3171382bc478d84a380dd958f8e08"
server_key_iv: "232dd1264fff84364e5b05fbba64a94b"
server_key_version: "encryptionkey@0"
I20260926 02:07:34.622913 25023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:34.646864 25023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:34.650924 25023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:34.651232 25023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:34.652285 25023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:34.653667 25023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:34.653746 25023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:34.653833 25023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:34.653927 25023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:34.675099 24883 heartbeater.cc:503] Master 127.18.159.190:45383 was elected leader, sending a full tablet report...
I20260926 02:07:34.688983 25023 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:38235
I20260926 02:07:34.689067 25150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:38235 every 8 connection(s)
I20260926 02:07:34.690338 25023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:07:34.694989 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 25023
I20260926 02:07:34.695127 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:07:34.695581 19070 external_mini_cluster.cc:1428] Setting key d4d93d39a8966da7ae89aaf7bfa5a422
Sep 26 02:07:34 dist-test-slave-03rn krb5kdc[24649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388454, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:34.717415 25153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:34.693112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:52485 (local address 127.18.159.190:45383)
0926 02:07:34.693644 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:34.693648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:34.694599 (+ 951us) server_negotiation.cc:407] Connection header received
0926 02:07:34.697097 (+ 2498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:34.697103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:34.697202 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:34.697327 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:34.699619 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:34.700568 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:34.702477 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:34.702975 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:34.703095 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:34.710991 (+ 7896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:34.710997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:34.711001 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:34.711037 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:34.714087 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:34.714897 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:34.714902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:34.714905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:34.715069 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:34.715516 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:34.715522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:34.715526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:34.715817 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:34.715990 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:34.716789 (+ 799us) server_negotiation.cc:299] Negotiation successful
0926 02:07:34.717022 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":259,"threads_started":1}
I20260926 02:07:34.718482 25154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:34.693626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:45383 (local address 127.18.159.131:52485)
0926 02:07:34.694381 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:34.694432 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:34.696684 (+ 2252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:34.697664 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:34.697679 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:34.698558 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:34.699418 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:34.699441 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:34.700729 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:34.700734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:34.702259 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:34.702284 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:34.703198 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:34.703203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:34.703419 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:34.707979 (+ 4560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:34.708012 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:34.710730 (+ 2718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:34.714253 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:34.714262 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:34.714277 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:34.714748 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:34.715188 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:34.715195 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:34.715203 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:34.715392 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:34.715959 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:34.715968 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:34.716415 (+ 447us) client_negotiation.cc:769] Sending connection context
0926 02:07:34.716701 (+ 286us) client_negotiation.cc:240] Negotiation successful
0926 02:07:34.717277 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":294,"threads_started":1}
I20260926 02:07:34.720091 25151 heartbeater.cc:347] Connected to a master server at 127.18.159.190:45383
I20260926 02:07:34.720492 25151 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:34.721261 25151 heartbeater.cc:511] Master 127.18.159.190:45383 requested a full tablet report, sending...
I20260926 02:07:34.723093 24694 ts_manager.cc:194] Registered new tserver with Master: 78a4771acb57465890b2c46b3dd7c756 (127.18.159.131:38235)
I20260926 02:07:34.724187 24694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:52485
I20260926 02:07:34.724676 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/test-admin.keytab.
Sep 26 02:07:34 dist-test-slave-03rn krb5kdc[24649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:34.754887 19070 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 02:07:34.760525 25159 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:34.757635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.18.159.129:42253)
0926 02:07:34.758317 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:34.758327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:34.758361 (+ 34us) server_negotiation.cc:407] Connection header received
0926 02:07:34.758580 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:34.758587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:34.758820 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:34.758993 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:34.760012 (+ 1019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34114: BlockingRecv error: recv got EOF from 127.0.0.1:34114 (error 108)
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":349,"threads_started":1}
W20260926 02:07:34.760605 19070 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/krb5cc' of type FILE
Sep 26 02:07:34 dist-test-slave-03rn krb5kdc[24649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:34.762292 19070 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 02:07:34 dist-test-slave-03rn krb5kdc[24649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:34.763660 19070 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 02:07:34 dist-test-slave-03rn krb5kdc[24649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:07:34.780793 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:34.765461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34118 (local address 127.18.159.129:42253)
0926 02:07:34.765666 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:34.765672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:34.765845 (+ 173us) server_negotiation.cc:407] Connection header received
0926 02:07:34.766053 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:34.766057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:34.766115 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:34.766219 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:34.767247 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:34.768312 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:34.769610 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:34.770186 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:34.770328 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:34.774021 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:34.774034 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:34.774048 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:34.774086 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:34.777490 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:34.778337 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:34.778346 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:34.778349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:34.778493 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:34.778954 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:34.778960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:34.778963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:34.779386 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:34.779719 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:34.780116 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 02:07:34.780454 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 02:07:34.787531 25159 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:34.785423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34122 (local address 127.18.159.129:42253)
0926 02:07:34.785594 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:34.785599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:34.786077 (+ 478us) server_negotiation.cc:407] Connection header received
0926 02:07:34.786212 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:34.786216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:34.786332 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:34.786444 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:34.787266 (+ 822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34122: BlockingRecv error: recv got EOF from 127.0.0.1:34122 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 02:07:34.787654 19070 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/krb5cc' of type FILE
Sep 26 02:07:34 dist-test-slave-03rn krb5kdc[24649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:34.789376 19070 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 02:07:34 dist-test-slave-03rn krb5kdc[24649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:34.790879 19070 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 02:07:34 dist-test-slave-03rn krb5kdc[24649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:07:34.806011 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:34.792529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.18.159.129:42253)
0926 02:07:34.792738 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:34.792744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:34.792969 (+ 225us) server_negotiation.cc:407] Connection header received
0926 02:07:34.793204 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:34.793209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:34.793270 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:34.793372 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:34.794384 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:34.795073 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:34.796473 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:34.796896 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:34.796996 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:34.800734 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:34.800740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:34.800744 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:34.800796 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:34.803492 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:34.804409 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:34.804413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:34.804415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:34.804544 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:34.804892 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:34.804896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:34.804898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:34.805170 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:34.805387 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:34.805608 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 02:07:34.805775 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 02:07:34.811403 25159 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:34.809197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34134 (local address 127.18.159.129:42253)
0926 02:07:34.809395 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:34.809399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:34.809769 (+ 370us) server_negotiation.cc:407] Connection header received
0926 02:07:34.809906 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:34.809911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:34.810000 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:34.810086 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:34.811127 (+ 1041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34134: BlockingRecv error: recv got EOF from 127.0.0.1:34134 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 02:07:34.811471 19070 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388402166281-19070-0/krb5kdc/krb5cc' of type FILE
Sep 26 02:07:34 dist-test-slave-03rn krb5kdc[24649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:34.813333 19070 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 02:07:34 dist-test-slave-03rn krb5kdc[24649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:34.815001 19070 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 02:07:34 dist-test-slave-03rn krb5kdc[24649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:07:34.830394 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:34.816594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34138 (local address 127.18.159.129:42253)
0926 02:07:34.816863 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:34.816869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:34.817140 (+ 271us) server_negotiation.cc:407] Connection header received
0926 02:07:34.817351 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:34.817369 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:34.817438 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:34.817544 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:34.818697 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:34.819495 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:34.820895 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:34.821338 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:34.821488 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:34.825409 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:34.825416 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:34.825420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:34.825461 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:34.827886 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:34.828609 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:34.828613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:34.828615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:34.828742 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:34.829088 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:34.829092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:34.829095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:34.829405 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:34.829632 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:34.829831 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 02:07:34.830074 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 02:07:34.832136 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 24755
I20260926 02:07:34.850714 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:34.832923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34140 (local address 127.18.159.129:42253)
0926 02:07:34.833079 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:34.833084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:34.833562 (+ 478us) server_negotiation.cc:407] Connection header received
0926 02:07:34.833739 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:34.833744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:34.833810 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:34.833939 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:34.834911 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:34.835630 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:34.837920 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:34.838666 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:34.838821 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:34.842319 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:34.842327 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:34.842332 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:34.842384 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:34.847129 (+ 4745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:34.848313 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:34.848322 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:34.848326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:34.848543 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:34.849094 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:34.849102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:34.849106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:34.849522 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:34.849809 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:34.850177 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 02:07:34.850384 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:07:34.852923 24878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:34.945968 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 24755
I20260926 02:07:34.962224 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 24889
Sep 26 02:07:34 dist-test-slave-03rn krb5kdc[24649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.130@KRBTEST.COM
I20260926 02:07:34.982769 25172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:34.963049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36766 (local address 127.18.159.130:38413)
0926 02:07:34.963605 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:34.963613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:34.963640 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:07:34.963707 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:34.963714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:34.963932 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:34.964075 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:34.965145 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:34.966224 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:34.967467 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:34.968051 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:34.968223 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:34.976085 (+ 7862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:34.976102 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:34.976118 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:34.976224 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:34.979371 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:34.980357 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:34.980370 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:34.980377 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:34.980554 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:34.981050 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:34.981062 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:34.981065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:34.981448 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:34.981773 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:34.982171 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 02:07:34.982468 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":300,"threads_started":1}
I20260926 02:07:34.984052 25012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:35.066036 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 24889
I20260926 02:07:35.081679 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 25023
Sep 26 02:07:35 dist-test-slave-03rn krb5kdc[24649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.131@KRBTEST.COM
I20260926 02:07:35.098148 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:35.082767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48810 (local address 127.18.159.131:38235)
0926 02:07:35.083486 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:35.083492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:35.083523 (+ 31us) server_negotiation.cc:407] Connection header received
0926 02:07:35.083581 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:35.083588 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:35.083800 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:35.083941 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:35.085050 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:35.086115 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:35.087401 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:35.088001 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:35.088104 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:35.092326 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:35.092339 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:35.092355 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:35.092402 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:35.095241 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:35.096006 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:35.096014 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:35.096016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:35.096143 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:35.096717 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:35.096720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:35.096722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:35.097085 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:35.097328 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:35.097552 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 02:07:35.097820 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":386,"threads_started":1}
I20260926 02:07:35.099459 25146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:35.182579 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 25023
I20260926 02:07:35.198264 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 24662
I20260926 02:07:35.199470 24724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:35.302661 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 24662
2026-09-26T02:07:35Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4130 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:07:35 dist-test-slave-03rn krb5kdc[25179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:07:35 dist-test-slave-03rn krb5kdc[25179](info): set up 2 sockets
Sep 26 02:07:35 dist-test-slave-03rn krb5kdc[25179](info): commencing operation
krb5kdc: starting...
W20260926 02:07:36.413404 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:07:36 dist-test-slave-03rn krb5kdc[25179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:07:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:07:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.190.keytab.
Entry for principal oryx/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.190.keytab.
I20260926 02:07:36.641844 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:35357
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.190.keytab
--principal=oryx/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:35357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:36.992367 25192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:36.992668 25192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:36.992765 25192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:37.000274 25192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:07:37.000397 25192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:37.000443 25192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:37.000492 25192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:07:37.000535 25192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:07:37.010665 25192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:35357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.190.keytab
--principal=oryx/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:35357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:37.012018 25192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:37.013681 25192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:37.026649 25197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:37.026582 25200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:37.026592 25198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:37.027631 25192 server_base.cc:1034] running on GCE node
I20260926 02:07:37.028590 25192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:37.030207 25192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:37.031473 25192 hybrid_clock.cc:648] HybridClock initialized: now 1790388457031404 us; error 62 us; skew 500 ppm
I20260926 02:07:37.031901 25192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:37 dist-test-slave-03rn krb5kdc[25179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388457, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:37.035320 25192 init.cc:377] Logged in from keytab as oryx/127.18.159.190@KRBTEST.COM (short username oryx)
I20260926 02:07:37.037079 25192 webserver.cc:460] Webserver started at http://127.18.159.190:42005/ using document root <none> and password file <none>
I20260926 02:07:37.037829 25192 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:37.037986 25192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:37.038307 25192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:37.041551 25192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "16268d1d4f86406b9faa8950ac12a76d"
format_stamp: "Formatted at 2026-09-26 02:07:37 on dist-test-slave-03rn"
server_key: "0508cc20642e5b21651a14b35fdc32e6"
server_key_iv: "7ee902a0a081af47bbb9d410140abf3b"
server_key_version: "encryptionkey@0"
I20260926 02:07:37.042450 25192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "16268d1d4f86406b9faa8950ac12a76d"
format_stamp: "Formatted at 2026-09-26 02:07:37 on dist-test-slave-03rn"
server_key: "0508cc20642e5b21651a14b35fdc32e6"
server_key_iv: "7ee902a0a081af47bbb9d410140abf3b"
server_key_version: "encryptionkey@0"
I20260926 02:07:37.048031 25192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:07:37.051753 25207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:37.053256 25192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:07:37.053473 25192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "16268d1d4f86406b9faa8950ac12a76d"
format_stamp: "Formatted at 2026-09-26 02:07:37 on dist-test-slave-03rn"
server_key: "0508cc20642e5b21651a14b35fdc32e6"
server_key_iv: "7ee902a0a081af47bbb9d410140abf3b"
server_key_version: "encryptionkey@0"
I20260926 02:07:37.053696 25192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:37.067647 25192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:37.072548 25192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:37.072860 25192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:37.092631 25192 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:35357
I20260926 02:07:37.092653 25258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:35357 every 8 connection(s)
I20260926 02:07:37.094153 25192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:07:37.098289 25259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:37.104734 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 25192
I20260926 02:07:37.104978 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:07:37.105613 19070 external_mini_cluster.cc:1428] Setting key 2f22e60a4e04710b4f303e9975f618cc
I20260926 02:07:37.110966 25259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d: Bootstrap starting.
I20260926 02:07:37.114930 25259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:37.116094 25259 log.cc:824] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:37.119272 25259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d: No bootstrap required, opened a new log
Sep 26 02:07:37 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM
I20260926 02:07:37.127825 25259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } }
I20260926 02:07:37.128324 25259 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:37.128479 25259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16268d1d4f86406b9faa8950ac12a76d, State: Initialized, Role: FOLLOWER
I20260926 02:07:37.129369 25259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } }
I20260926 02:07:37.129668 25259 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:07:37.129874 25259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:07:37.130079 25259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:37.130082 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:37.107948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46794 (local address 127.18.159.190:35357)
0926 02:07:37.108871 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:37.108888 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:37.108934 (+ 46us) server_negotiation.cc:407] Connection header received
0926 02:07:37.110666 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:37.110693 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:37.111234 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:37.111675 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:37.113446 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:37.114787 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:37.116241 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:37.117112 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:37.117355 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:37.121673 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:37.121684 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:37.121704 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:37.121752 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:37.125282 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:37.126132 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:37.126140 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:37.126147 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:37.126342 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:37.126784 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:37.126791 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:37.126794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:37.127429 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:37.127764 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:37.128285 (+ 521us) server_negotiation.cc:299] Negotiation successful
0926 02:07:37.128812 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":411,"threads_started":1}
I20260926 02:07:37.133104 25259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } }
I20260926 02:07:37.133769 25259 leader_election.cc:302] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16268d1d4f86406b9faa8950ac12a76d; no voters:
I20260926 02:07:37.134435 25259 leader_election.cc:288] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:07:37.134588 25264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:37.135449 25264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [term 1 LEADER]: Becoming Leader. State: Replica: 16268d1d4f86406b9faa8950ac12a76d, State: Running, Role: LEADER
I20260926 02:07:37.136052 25264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } }
I20260926 02:07:37.137192 25259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:07:37.139143 25265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } }
I20260926 02:07:37.139526 25265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:37.141417 25266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16268d1d4f86406b9faa8950ac12a76d. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } }
I20260926 02:07:37.141863 25269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:07:37.142024 25266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:37.147014 25269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:07:37.158602 25269 catalog_manager.cc:1347] Generated new cluster ID: 1fd63064f7554b2a889942cd3cb66892
I20260926 02:07:37.158738 25269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:07:37.178879 25269 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:07:37.181025 25269 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:07:37.194641 25269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d: Generated new TSK 0
I20260926 02:07:37.195814 25269 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.159.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.159.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.159.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.129.keytab.
Entry for principal oryx/127.18.159.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.129.keytab.
I20260926 02:07:37.265377 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35357
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.129.keytab
--principal=oryx/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:37.614528 25285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:37.614840 25285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:37.615022 25285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:37.622566 25285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:37.622705 25285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:37.622936 25285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:07:37.633751 25285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.129.keytab
--principal=oryx/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:37.635082 25285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:37.636523 25285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:37.650992 25290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:37.651103 25291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:37.650974 25293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:37.658006 25285 server_base.cc:1034] running on GCE node
I20260926 02:07:37.658838 25285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:37.660233 25285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:37.661525 25285 hybrid_clock.cc:648] HybridClock initialized: now 1790388457661454 us; error 75 us; skew 500 ppm
I20260926 02:07:37.661970 25285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:37 dist-test-slave-03rn krb5kdc[25179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388457, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:37.665757 25285 init.cc:377] Logged in from keytab as oryx/127.18.159.129@KRBTEST.COM (short username oryx)
I20260926 02:07:37.667498 25285 webserver.cc:460] Webserver started at http://127.18.159.129:46123/ using document root <none> and password file <none>
I20260926 02:07:37.668427 25285 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:37.668586 25285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:37.668928 25285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:37.672063 25285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "3f30c24fe8b7457ba6f65e5ef22fee66"
format_stamp: "Formatted at 2026-09-26 02:07:37 on dist-test-slave-03rn"
server_key: "338e66e6369f9413d9bc8cead76fcf1d"
server_key_iv: "399888b8df1bccb781147b21dc5a9b49"
server_key_version: "encryptionkey@0"
I20260926 02:07:37.672991 25285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "3f30c24fe8b7457ba6f65e5ef22fee66"
format_stamp: "Formatted at 2026-09-26 02:07:37 on dist-test-slave-03rn"
server_key: "338e66e6369f9413d9bc8cead76fcf1d"
server_key_iv: "399888b8df1bccb781147b21dc5a9b49"
server_key_version: "encryptionkey@0"
I20260926 02:07:37.678745 25285 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:07:37.682631 25300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:37.684438 25285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:07:37.684643 25285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "3f30c24fe8b7457ba6f65e5ef22fee66"
format_stamp: "Formatted at 2026-09-26 02:07:37 on dist-test-slave-03rn"
server_key: "338e66e6369f9413d9bc8cead76fcf1d"
server_key_iv: "399888b8df1bccb781147b21dc5a9b49"
server_key_version: "encryptionkey@0"
I20260926 02:07:37.684868 25285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:37.698999 25285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:37.704133 25285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:37.704516 25285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:37.705590 25285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:37.707240 25285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:37.707345 25285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:37.707499 25285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:37.707571 25285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:37.742411 25285 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:34533
I20260926 02:07:37.742470 25412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:34533 every 8 connection(s)
I20260926 02:07:37.743985 25285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:07:37.748090 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 25285
I20260926 02:07:37.748368 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:07:37.749310 19070 external_mini_cluster.cc:1428] Setting key 19a44ccc1cb5be39f396a6c0fd45e537
Sep 26 02:07:37 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388457, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM
I20260926 02:07:37.769366 25415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:37.746886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:60381 (local address 127.18.159.190:35357)
0926 02:07:37.747369 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:37.747374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:37.748573 (+ 1199us) server_negotiation.cc:407] Connection header received
0926 02:07:37.750628 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:37.750636 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:37.750719 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:37.750851 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:37.753049 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:37.753822 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:37.756533 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:37.757067 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:37.757211 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:37.761481 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:37.761491 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:37.761500 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:37.761552 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:37.765536 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:37.766416 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:37.766425 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:37.766432 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:37.766596 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:37.767063 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:37.767072 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:37.767078 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:37.767515 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:37.767678 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:37.768495 (+ 817us) server_negotiation.cc:299] Negotiation successful
0926 02:07:37.768815 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":252,"threads_started":1}
I20260926 02:07:37.769979 25416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:37.747506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.129:60381)
0926 02:07:37.748366 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:37.748422 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:37.750351 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:37.751062 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:37.751070 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:37.751871 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:37.752786 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:37.752816 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:37.754017 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:37.754025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:37.756341 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:37.756362 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:37.757352 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:37.757362 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:37.757721 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:37.758833 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:37.758859 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:37.761162 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:37.765735 (+ 4573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:37.765744 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:37.765760 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:37.766246 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:37.766716 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:37.766719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:37.766722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:37.766919 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:37.767641 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:37.767647 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:37.768011 (+ 364us) client_negotiation.cc:769] Sending connection context
0926 02:07:37.768310 (+ 299us) client_negotiation.cc:240] Negotiation successful
0926 02:07:37.768765 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":351,"threads_started":1}
I20260926 02:07:37.771718 25413 heartbeater.cc:347] Connected to a master server at 127.18.159.190:35357
I20260926 02:07:37.772110 25413 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:37.773216 25413 heartbeater.cc:511] Master 127.18.159.190:35357 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.18.159.130@KRBTEST.COM; defaulting to no policy
I20260926 02:07:37.775892 25224 ts_manager.cc:194] Registered new tserver with Master: 3f30c24fe8b7457ba6f65e5ef22fee66 (127.18.159.129:34533)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.159.130@KRBTEST.COM" created.
I20260926 02:07:37.777767 25224 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:60381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.159.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.130.keytab.
Entry for principal oryx/127.18.159.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.130.keytab.
I20260926 02:07:37.804127 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35357
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.130.keytab
--principal=oryx/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:38.151743 25419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:38.152055 25419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:38.152237 25419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:38.160142 25419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:38.160292 25419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:38.160498 25419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:07:38.170573 25419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.130.keytab
--principal=oryx/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:38.171975 25419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:38.173363 25419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:38.187109 25424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:38.187004 25425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:38.187032 25427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:38.188249 25419 server_base.cc:1034] running on GCE node
I20260926 02:07:38.188952 25419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:38.190011 25419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:38.191258 25419 hybrid_clock.cc:648] HybridClock initialized: now 1790388458191191 us; error 64 us; skew 500 ppm
I20260926 02:07:38.191610 25419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:38 dist-test-slave-03rn krb5kdc[25179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388458, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:38.195202 25419 init.cc:377] Logged in from keytab as oryx/127.18.159.130@KRBTEST.COM (short username oryx)
I20260926 02:07:38.196882 25419 webserver.cc:460] Webserver started at http://127.18.159.130:40743/ using document root <none> and password file <none>
I20260926 02:07:38.197597 25419 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:38.197762 25419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:38.198071 25419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:38.201202 25419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "17e22f81cc54435fa3db75d74f44e284"
format_stamp: "Formatted at 2026-09-26 02:07:38 on dist-test-slave-03rn"
server_key: "2c60bc3b2c151ae6045c866f9a434fc4"
server_key_iv: "69f0cd51b505f08d95d83e92e7a95f14"
server_key_version: "encryptionkey@0"
I20260926 02:07:38.202039 25419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "17e22f81cc54435fa3db75d74f44e284"
format_stamp: "Formatted at 2026-09-26 02:07:38 on dist-test-slave-03rn"
server_key: "2c60bc3b2c151ae6045c866f9a434fc4"
server_key_iv: "69f0cd51b505f08d95d83e92e7a95f14"
server_key_version: "encryptionkey@0"
I20260926 02:07:38.207536 25419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:07:38.211251 25434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:38.212697 25419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:07:38.212895 25419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "17e22f81cc54435fa3db75d74f44e284"
format_stamp: "Formatted at 2026-09-26 02:07:38 on dist-test-slave-03rn"
server_key: "2c60bc3b2c151ae6045c866f9a434fc4"
server_key_iv: "69f0cd51b505f08d95d83e92e7a95f14"
server_key_version: "encryptionkey@0"
I20260926 02:07:38.213109 25419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:38.235424 25419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:38.240124 25419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:38.240504 25419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:38.241580 25419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:38.243048 25419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:38.243162 25419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:38.243310 25419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:38.243382 25419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:38.279346 25419 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:42645
I20260926 02:07:38.279407 25546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:42645 every 8 connection(s)
I20260926 02:07:38.280726 25419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:07:38.286404 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 25419
I20260926 02:07:38.286598 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:07:38.287055 19070 external_mini_cluster.cc:1428] Setting key 064a9611063f30cc2e76ac45b06965ee
Sep 26 02:07:38 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388458, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM
I20260926 02:07:38.304380 25549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:38.283478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:47725 (local address 127.18.159.190:35357)
0926 02:07:38.283947 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:38.283954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:38.285130 (+ 1176us) server_negotiation.cc:407] Connection header received
0926 02:07:38.287175 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:38.287183 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:38.287282 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:38.287415 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:38.289500 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:38.290683 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:38.292460 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:38.293219 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:38.293369 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:38.297608 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:38.297613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:38.297616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:38.297652 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:38.300514 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:38.301630 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:38.301634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:38.301637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:38.301772 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:38.302364 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:38.302367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:38.302369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:38.302633 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:38.302749 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:38.303776 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0926 02:07:38.304004 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":206,"threads_started":1}
I20260926 02:07:38.306046 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:38.284107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.130:47725)
0926 02:07:38.284936 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:38.284991 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:38.286820 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:38.287695 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:38.287708 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:38.288495 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:38.289264 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:38.289308 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:38.290865 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:38.290869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:38.292303 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:38.292316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:38.293456 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:38.293461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:38.293694 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:38.294442 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:38.294473 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:38.297291 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:38.300703 (+ 3412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:38.300716 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:38.300741 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:38.301475 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:38.301914 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:38.301922 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:38.301929 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:38.302232 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:38.302776 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:38.302787 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:38.303341 (+ 554us) client_negotiation.cc:769] Sending connection context
0926 02:07:38.303670 (+ 329us) client_negotiation.cc:240] Negotiation successful
0926 02:07:38.304377 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":306,"threads_started":1}
I20260926 02:07:38.307674 25547 heartbeater.cc:347] Connected to a master server at 127.18.159.190:35357
I20260926 02:07:38.308005 25547 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:38.308813 25547 heartbeater.cc:511] Master 127.18.159.190:35357 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.18.159.131@KRBTEST.COM; defaulting to no policy
I20260926 02:07:38.310401 25224 ts_manager.cc:194] Registered new tserver with Master: 17e22f81cc54435fa3db75d74f44e284 (127.18.159.130:42645)
I20260926 02:07:38.311239 25224 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:47725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.159.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.159.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.131.keytab.
Entry for principal oryx/127.18.159.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.131.keytab.
I20260926 02:07:38.339699 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35357
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.131.keytab
--principal=oryx/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:38.685141 25553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:38.685441 25553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:38.685604 25553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:38.693359 25553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:38.693478 25553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:38.693656 25553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:07:38.704101 25553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.131.keytab
--principal=oryx/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:38.706303 25553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:38.707957 25553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:38.724186 25558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:38.724287 25561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:38.724604 25553 server_base.cc:1034] running on GCE node
W20260926 02:07:38.724287 25559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:38.725593 25553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:38.726750 25553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:38.728021 25553 hybrid_clock.cc:648] HybridClock initialized: now 1790388458727956 us; error 57 us; skew 500 ppm
I20260926 02:07:38.728443 25553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:38 dist-test-slave-03rn krb5kdc[25179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388458, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:38.731983 25553 init.cc:377] Logged in from keytab as oryx/127.18.159.131@KRBTEST.COM (short username oryx)
I20260926 02:07:38.733812 25553 webserver.cc:460] Webserver started at http://127.18.159.131:38061/ using document root <none> and password file <none>
I20260926 02:07:38.734599 25553 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:38.734761 25553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:38.735086 25553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:38.738338 25553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c"
format_stamp: "Formatted at 2026-09-26 02:07:38 on dist-test-slave-03rn"
server_key: "e8b468411bd4dbc8a0529a876a9ea6b4"
server_key_iv: "16709f93faccc67d7db705f248f908d7"
server_key_version: "encryptionkey@0"
I20260926 02:07:38.739545 25553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c"
format_stamp: "Formatted at 2026-09-26 02:07:38 on dist-test-slave-03rn"
server_key: "e8b468411bd4dbc8a0529a876a9ea6b4"
server_key_iv: "16709f93faccc67d7db705f248f908d7"
server_key_version: "encryptionkey@0"
I20260926 02:07:38.745997 25553 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260926 02:07:38.751453 25568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:38.753599 25553 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 02:07:38.753851 25553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c"
format_stamp: "Formatted at 2026-09-26 02:07:38 on dist-test-slave-03rn"
server_key: "e8b468411bd4dbc8a0529a876a9ea6b4"
server_key_iv: "16709f93faccc67d7db705f248f908d7"
server_key_version: "encryptionkey@0"
I20260926 02:07:38.754117 25553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:38.768749 25553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:38.773626 25553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:38.773952 25553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:38.775098 25553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:38.776774 25553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:38.776901 25553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:38.777086 25553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:38.777168 25553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:38.781802 25413 heartbeater.cc:503] Master 127.18.159.190:35357 was elected leader, sending a full tablet report...
I20260926 02:07:38.815068 25553 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:41509
I20260926 02:07:38.815115 25680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:41509 every 8 connection(s)
I20260926 02:07:38.816620 25553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:07:38.824237 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 25553
I20260926 02:07:38.824457 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:07:38.824934 19070 external_mini_cluster.cc:1428] Setting key c29e426b31fef1e28a78b0ad40b48c9e
Sep 26 02:07:38 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388458, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM
I20260926 02:07:38.842415 25683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:38.820375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:59447 (local address 127.18.159.190:35357)
0926 02:07:38.820793 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:38.820798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:38.822001 (+ 1203us) server_negotiation.cc:407] Connection header received
0926 02:07:38.823983 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:38.823988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:38.824108 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:38.824297 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:38.826585 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:38.827908 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:38.830034 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:38.830461 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:38.830569 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:38.834473 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:38.834478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:38.834482 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:38.834519 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:38.837630 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:38.838723 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:38.838729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:38.838732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:38.838919 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:38.839631 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:38.839638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:38.839641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:38.840432 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:38.840644 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:38.841731 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0926 02:07:38.841981 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":214,"threads_started":1}
I20260926 02:07:38.843523 25684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:38.821010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.131:59447)
0926 02:07:38.821784 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:38.821833 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:38.823608 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:38.824608 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:38.824629 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:38.825452 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:38.826371 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:38.826405 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:38.828228 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:38.828234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:38.829852 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:38.829869 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:38.830704 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:38.830709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:38.830980 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:38.831678 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:38.831701 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:38.834150 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:38.837860 (+ 3710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:38.837870 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:38.837887 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:38.838514 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:38.839128 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:38.839132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:38.839135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:38.839365 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:38.840627 (+ 1262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:38.840639 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:38.841108 (+ 469us) client_negotiation.cc:769] Sending connection context
0926 02:07:38.841481 (+ 373us) client_negotiation.cc:240] Negotiation successful
0926 02:07:38.842099 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":294,"threads_started":1}
I20260926 02:07:38.845046 25681 heartbeater.cc:347] Connected to a master server at 127.18.159.190:35357
I20260926 02:07:38.845418 25681 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:38.846233 25681 heartbeater.cc:511] Master 127.18.159.190:35357 requested a full tablet report, sending...
I20260926 02:07:38.848023 25224 ts_manager.cc:194] Registered new tserver with Master: f9a809af0ebb4c5b8b485259aeb2fb1c (127.18.159.131:41509)
I20260926 02:07:38.848945 25224 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:59447
I20260926 02:07:38.855525 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.159.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.159.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.159.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.189.keytab.
Entry for principal oryx/127.18.159.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.189.keytab.
I20260926 02:07:38.912330 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.189:43111
--webserver_interface=127.18.159.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.189.keytab
--principal=oryx/127.18.159.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:35357,127.18.159.189:43111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:39.269940 25687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:39.270319 25687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:39.270473 25687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:39.278064 25687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:07:39.278227 25687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:39.278337 25687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:39.278455 25687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:07:39.278532 25687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:07:39.288970 25687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:35357,127.18.159.189:43111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.189.keytab
--principal=oryx/127.18.159.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.189:43111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.159.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
W20260926 02:07:39.289422 25687 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.18.159.190:35357,127.18.159.189:43111'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 02:07:39.290491 25687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:39.291844 25687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:39.304525 25692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:39.304611 25693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:39.304584 25695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:39.306376 25687 server_base.cc:1034] running on GCE node
I20260926 02:07:39.307299 25687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:39.308971 25687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:39.310226 25687 hybrid_clock.cc:648] HybridClock initialized: now 1790388459310148 us; error 68 us; skew 500 ppm
I20260926 02:07:39.310585 25687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:39 dist-test-slave-03rn krb5kdc[25179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388459, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:39.314030 25687 init.cc:377] Logged in from keytab as oryx/127.18.159.189@KRBTEST.COM (short username oryx)
I20260926 02:07:39.314575 25547 heartbeater.cc:503] Master 127.18.159.190:35357 was elected leader, sending a full tablet report...
I20260926 02:07:39.315858 25687 webserver.cc:460] Webserver started at http://127.18.159.189:43565/ using document root <none> and password file <none>
I20260926 02:07:39.316813 25687 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:39.316915 25687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:39.317214 25687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:39.320525 25687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/data/instance:
uuid: "4d7e2176af0a475eab3956ac474f6320"
format_stamp: "Formatted at 2026-09-26 02:07:39 on dist-test-slave-03rn"
server_key: "a1551a08d10c87af79b68260efc3cd55"
server_key_iv: "3522c9796a1fe2ae667ef38a72c27e49"
server_key_version: "encryptionkey@0"
I20260926 02:07:39.321337 25687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/wal/instance:
uuid: "4d7e2176af0a475eab3956ac474f6320"
format_stamp: "Formatted at 2026-09-26 02:07:39 on dist-test-slave-03rn"
server_key: "a1551a08d10c87af79b68260efc3cd55"
server_key_iv: "3522c9796a1fe2ae667ef38a72c27e49"
server_key_version: "encryptionkey@0"
I20260926 02:07:39.326807 25687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:07:39.330736 25702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:39.332432 25687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:07:39.332623 25687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/wal
uuid: "4d7e2176af0a475eab3956ac474f6320"
format_stamp: "Formatted at 2026-09-26 02:07:39 on dist-test-slave-03rn"
server_key: "a1551a08d10c87af79b68260efc3cd55"
server_key_iv: "3522c9796a1fe2ae667ef38a72c27e49"
server_key_version: "encryptionkey@0"
I20260926 02:07:39.332885 25687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:39.350942 25687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:39.355937 25687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:39.356264 25687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:39.376075 25687 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.189:43111
I20260926 02:07:39.376071 25753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.189:43111 every 8 connection(s)
I20260926 02:07:39.377593 25687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/data/info.pb
I20260926 02:07:39.381858 25754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:39.386572 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 25687
I20260926 02:07:39.390357 25754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:07:39 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.159.189@KRBTEST.COM
Sep 26 02:07:39 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388459, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.189@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM
I20260926 02:07:39.412690 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:39.394413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46802 (local address 127.18.159.190:35357)
0926 02:07:39.394964 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:39.394971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:39.395533 (+ 562us) server_negotiation.cc:407] Connection header received
0926 02:07:39.395723 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:39.395727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:39.395802 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:39.395906 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:39.397349 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.398401 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.400668 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.401356 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.401515 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:39.405711 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:39.405718 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:39.405722 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:39.405777 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:39.409347 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:39.410291 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:39.410298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:39.410303 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:39.410449 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:39.410858 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:39.410863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:39.410866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:39.411291 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:39.411425 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:39.412094 (+ 669us) server_negotiation.cc:299] Negotiation successful
0926 02:07:39.412357 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":259,"threads_started":1}
I20260926 02:07:39.413650 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:39.389111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.18.159.189:43111)
0926 02:07:39.389960 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:39.389973 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:39.390018 (+ 45us) server_negotiation.cc:407] Connection header received
0926 02:07:39.391625 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:39.391652 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:39.392219 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:39.392701 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:39.394289 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.396539 (+ 2250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.398245 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.399483 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.399713 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:39.404706 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:39.404715 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:39.404730 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:39.404763 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:39.408841 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:39.409694 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:39.409702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:39.409704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:39.409866 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:39.410348 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:39.410353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:39.410356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:39.411102 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:39.411421 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:39.411909 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 02:07:39.412418 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":637,"spinlock_wait_cycles":22784,"thread_start_us":397,"threads_started":1}
I20260926 02:07:39.413822 25759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:39.394441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.0.0.1:46802)
0926 02:07:39.395363 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:39.395415 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:39.395583 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:39.396009 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:39.396017 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:39.396917 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:39.397178 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:39.397187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.398580 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.398588 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:39.400510 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:39.400523 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.401637 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.401642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:39.401821 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:39.402583 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:39.402612 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:39.405453 (+ 2841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:39.409490 (+ 4037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:39.409503 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:39.409516 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:39.410137 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:39.410561 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:39.410564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:39.410566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:39.410711 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:39.411429 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:39.411440 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:39.411669 (+ 229us) client_negotiation.cc:769] Sending connection context
0926 02:07:39.411963 (+ 294us) client_negotiation.cc:240] Negotiation successful
0926 02:07:39.412545 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":122,"thread_start_us":478,"threads_started":1}
I20260926 02:07:39.415771 25754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:07:39 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388459, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.189@KRBTEST.COM for oryx/127.18.159.189@KRBTEST.COM
I20260926 02:07:39.430050 25759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:39.416718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:43111 (local address 127.0.0.1:37778)
0926 02:07:39.416984 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:39.417008 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:39.417111 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:39.417411 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:39.417415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:39.417842 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:39.418037 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:39.418052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.419096 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.419100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:39.420026 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:39.420038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.420948 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.420953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:39.421131 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:39.421803 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:39.421823 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:39.424234 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:39.427601 (+ 3367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:39.427606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:39.427609 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:39.428069 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:39.428549 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:39.428553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:39.428555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:39.428700 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:39.429251 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:39.429255 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:39.429384 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 02:07:39.429508 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 02:07:39.429805 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260926 02:07:39.430291 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:39.416585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37778 (local address 127.18.159.189:43111)
0926 02:07:39.416784 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:39.416790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:39.417081 (+ 291us) server_negotiation.cc:407] Connection header received
0926 02:07:39.417205 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:39.417208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:39.417262 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:39.417358 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:39.418212 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.418938 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.420265 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.420716 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.420837 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:39.424472 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:39.424477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:39.424481 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:39.424525 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:39.427407 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:39.428292 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:39.428296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:39.428299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:39.428419 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:39.428802 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:39.428805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:39.428807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:39.429114 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:39.429230 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:39.429655 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 02:07:39.429954 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":65}
I20260926 02:07:39.435205 25754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: Bootstrap starting.
I20260926 02:07:39.438997 25754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:39.440230 25754 log.cc:824] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:39.443188 25754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: No bootstrap required, opened a new log
I20260926 02:07:39.450641 25754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } }
I20260926 02:07:39.451074 25754 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:39.451206 25754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d7e2176af0a475eab3956ac474f6320, State: Initialized, Role: FOLLOWER
I20260926 02:07:39.451973 25754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } }
I20260926 02:07:39.454097 25761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } } }
I20260926 02:07:39.454591 25761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:07:39.455621 25754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 02:07:39.466212 25772 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:07:39.466461 25772 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:07:39.475414 25687 master_runner.cc:428] Detected that this master 4d7e2176af0a475eab3956ac474f6320 is joining an existing cluster
I20260926 02:07:39.475584 25687 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.159.189:43111
WARNING: no policy specified for oryx/127.18.159.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.159.188@KRBTEST.COM" created.
I20260926 02:07:39.516777 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:39.480851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55576 (local address 127.18.159.190:35357)
0926 02:07:39.481127 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:39.481135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:39.495673 (+ 14538us) server_negotiation.cc:407] Connection header received
0926 02:07:39.495813 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:39.495818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:39.495923 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:39.496043 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:39.499497 (+ 3454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.500813 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.503592 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.504523 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.504661 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:39.507481 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:39.507487 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:39.507490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:39.507526 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:39.511647 (+ 4121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:39.514021 (+ 2374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:39.514028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:39.514033 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:39.514214 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:39.514724 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:39.514733 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:39.514740 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:39.515144 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:39.515269 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:39.516112 (+ 843us) server_negotiation.cc:299] Negotiation successful
0926 02:07:39.516401 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 02:07:39.516798 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:39.480542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.18.159.189:43111)
0926 02:07:39.481152 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:39.481161 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:39.495358 (+ 14197us) server_negotiation.cc:407] Connection header received
0926 02:07:39.495549 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:39.495553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:39.495657 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:39.495810 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:39.499190 (+ 3380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.500528 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.503024 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.503875 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.504003 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:39.508207 (+ 4204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:39.508216 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:39.508223 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:39.508266 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:39.513206 (+ 4940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:39.514148 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:39.514153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:39.514157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:39.514337 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:39.514755 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:39.514760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:39.514763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:39.515140 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:39.515269 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:39.516013 (+ 744us) server_negotiation.cc:299] Negotiation successful
0926 02:07:39.516299 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427}
I20260926 02:07:39.517938 25779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:39.481553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:43111 (local address 127.0.0.1:37794)
0926 02:07:39.495175 (+ 13622us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:39.495204 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:39.495414 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:39.495861 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:39.495867 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:39.497448 (+ 1581us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:39.498960 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:39.498980 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.500790 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.500796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:39.502814 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:39.502832 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.504444 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.504451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:39.504639 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:39.505538 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:39.505567 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:39.507319 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:39.513373 (+ 6054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:39.513378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:39.513383 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:39.514027 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:39.514492 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:39.514496 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:39.514500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:39.514651 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:39.515369 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:39.515373 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:39.515707 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 02:07:39.517000 (+ 1293us) client_negotiation.cc:240] Negotiation successful
0926 02:07:39.517335 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13490,"mutex_wait_us":60,"thread_start_us":12607,"threads_started":1}
I20260926 02:07:39.519326 25780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:39.480316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.0.0.1:55576)
0926 02:07:39.494615 (+ 14299us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:39.495565 (+ 950us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:39.495678 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:39.496304 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:39.496307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:39.497649 (+ 1342us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:39.499246 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:39.499267 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.501054 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.501063 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:39.503357 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:39.503376 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.505067 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.505075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:39.505338 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:39.506027 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:39.506058 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:39.507217 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:39.511826 (+ 4609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:39.511834 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:39.511841 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:39.513829 (+ 1988us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:39.514377 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:39.514385 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:39.514391 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:39.514578 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:39.515364 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:39.515369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:39.515815 (+ 446us) client_negotiation.cc:769] Sending connection context
0926 02:07:39.517603 (+ 1788us) client_negotiation.cc:240] Negotiation successful
0926 02:07:39.518828 (+ 1225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14035,"mutex_wait_us":1149,"spinlock_wait_cycles":71552,"thread_start_us":14471,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.159.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.188.keytab.
Entry for principal oryx/127.18.159.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.188.keytab.
I20260926 02:07:39.535595 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.188:40445
--webserver_interface=127.18.159.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.188.keytab
--principal=oryx/127.18.159.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:35357,127.18.159.189:43111,127.18.159.188:40445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:07:39 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388457, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.190@KRBTEST.COM for oryx/127.18.159.189@KRBTEST.COM
I20260926 02:07:39.543735 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:39.525914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.18.159.189:43111)
0926 02:07:39.526116 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:39.526123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:39.526956 (+ 833us) server_negotiation.cc:407] Connection header received
0926 02:07:39.527409 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:39.527416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:39.527541 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:39.527687 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:39.529012 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.530390 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.531938 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.532884 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.533032 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:39.537091 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:39.537096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:39.537100 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:39.537136 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:39.540501 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:39.541378 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:39.541385 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:39.541389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:39.541564 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:39.542004 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:39.542014 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:39.542019 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:39.542310 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:39.542485 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:39.543032 (+ 547us) server_negotiation.cc:299] Negotiation successful
0926 02:07:39.543321 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:07:39.543792 25782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:39.526065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:43111 (local address 127.0.0.1:37810)
0926 02:07:39.526797 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:39.526855 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:39.527133 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:39.527749 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:39.527760 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:39.528565 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:39.528860 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:39.528876 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.530553 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.530563 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:39.531786 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:39.531800 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.533062 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.533067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:39.533276 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:39.534126 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:39.534156 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:39.536802 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:39.540690 (+ 3888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:39.540709 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:39.540718 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:39.541231 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:39.541700 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:39.541706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:39.541710 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:39.541876 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:39.542465 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:39.542475 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:39.542700 (+ 225us) client_negotiation.cc:769] Sending connection context
0926 02:07:39.542931 (+ 231us) client_negotiation.cc:240] Negotiation successful
0926 02:07:39.543318 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":336,"threads_started":1}
I20260926 02:07:39.546332 25224 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.18.159.189:43111: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4d7e2176af0a475eab3956ac474f6320"
member_type: NON_VOTER
last_known_addr {
host: "127.18.159.189"
port: 43111
}
attrs {
promote: true
}
}
dest_uuid: "16268d1d4f86406b9faa8950ac12a76d"
cas_config_opid_index: -1
I20260926 02:07:39.547453 25224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: NON_VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: true } }
I20260926 02:07:39.564107 25729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:39.566546 25786 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4d7e2176af0a475eab3956ac474f6320 (127.18.159.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: NON_VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: true } } }
I20260926 02:07:39.568048 25729 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 1 FOLLOWER]: Refusing update from remote peer 16268d1d4f86406b9faa8950ac12a76d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 02:07:39.569494 25785 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: NON_VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:39.573388 25786 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.18.159.189:43111
I20260926 02:07:39.574095 25687 master.cc:537] Master@127.18.159.189:43111 shutting down...
I20260926 02:07:39.574290 25785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: NON_VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: true } } }
I20260926 02:07:39.574563 25785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:39.597958 25729 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4d7e2176af0a475eab3956ac474f6320 (127.18.159.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: NON_VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: true } } }
I20260926 02:07:39.621472 25794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: NON_VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: true } } }
I20260926 02:07:39.621920 25794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: This master's current role is: LEARNER
I20260926 02:07:39.622006 25790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: NON_VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: true } } }
I20260926 02:07:39.622360 25790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: This master's current role is: LEARNER
I20260926 02:07:39.622354 25761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16268d1d4f86406b9faa8950ac12a76d. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: NON_VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: true } } }
I20260926 02:07:39.622575 25761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: This master's current role is: LEARNER
I20260926 02:07:39.625588 25687 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 1 LEARNER]: Raft consensus shutting down.
I20260926 02:07:39.626153 25687 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 1 LEARNER]: Raft consensus is shut down!
I20260926 02:07:39.626325 25687 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: stopping tablet replica
I20260926 02:07:39.634580 25687 master.cc:552] Master@127.18.159.189:43111 shutdown complete.
I20260926 02:07:39.634738 25687 master_runner.cc:305] Clearing existing system tablet
I20260926 02:07:39.635210 25687 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:39.635356 25687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:39.638700 25687 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260926 02:07:39.640642 25796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:39.641048 25687 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:39.641530 25687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 02:07:39.641682 25687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/wal
uuid: "4d7e2176af0a475eab3956ac474f6320"
format_stamp: "Formatted at 2026-09-26 02:07:39 on dist-test-slave-03rn"
server_key: "a1551a08d10c87af79b68260efc3cd55"
server_key_iv: "3522c9796a1fe2ae667ef38a72c27e49"
server_key_version: "encryptionkey@0"
I20260926 02:07:39.647151 25687 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:39.654451 25687 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 02:07:39.654815 25687 log.cc:1196] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 02:07:39.655556 25687 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: Deleting consensus metadata
I20260926 02:07:39.655910 25687 master_runner.cc:315] Copying system tablet from 127.18.159.190:35357
I20260926 02:07:39.665550 25687 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: tablet copy: Beginning tablet copy session from remote peer at address 127.18.159.190:35357
I20260926 02:07:39.688146 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:39.666646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55588 (local address 127.18.159.190:35357)
0926 02:07:39.666886 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:39.666894 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:39.672947 (+ 6053us) server_negotiation.cc:407] Connection header received
0926 02:07:39.673155 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:39.673161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:39.673250 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:39.673384 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:39.674539 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.675732 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.677728 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.678566 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.678721 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:39.681174 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:39.681181 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:39.681186 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:39.681227 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:39.684875 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:39.685753 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:39.685760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:39.685763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:39.685946 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:39.686407 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:39.686412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:39.686416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:39.686807 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:39.686966 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:39.687505 (+ 539us) server_negotiation.cc:299] Negotiation successful
0926 02:07:39.687746 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:07:39.688215 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:39.666574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.0.0.1:55588)
0926 02:07:39.672758 (+ 6184us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:39.672782 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:39.672994 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:39.673445 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:39.673452 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:39.674034 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:39.674333 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:39.674350 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.675944 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.675951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:39.677548 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:39.677564 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:39.678787 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:39.678794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:39.678992 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:39.679810 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:39.679841 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:39.680934 (+ 1093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:39.685047 (+ 4113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:39.685053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:39.685057 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:39.685596 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:39.686090 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:39.686098 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:39.686104 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:39.686277 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:39.686952 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:39.686956 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:39.687166 (+ 210us) client_negotiation.cc:769] Sending connection context
0926 02:07:39.687358 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 02:07:39.687730 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6024,"thread_start_us":5858,"threads_started":1}
I20260926 02:07:39.689921 25244 tablet_copy_service.cc:140] P 16268d1d4f86406b9faa8950ac12a76d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4d7e2176af0a475eab3956ac474f6320 ({username='oryx', principal='oryx/127.18.159.189@KRBTEST.COM'} at 127.0.0.1:55588)
I20260926 02:07:39.690290 25244 tablet_copy_service.cc:161] P 16268d1d4f86406b9faa8950ac12a76d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4d7e2176af0a475eab3956ac474f6320 at {username='oryx', principal='oryx/127.18.159.189@KRBTEST.COM'} at 127.0.0.1:55588: session id = 4d7e2176af0a475eab3956ac474f6320-00000000000000000000000000000000
I20260926 02:07:39.694819 25244 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 02:07:39.697332 25687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 02:07:39.704586 25687 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: tablet copy: Starting download of 0 data blocks...
I20260926 02:07:39.704965 25687 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: tablet copy: Starting download of 1 WAL segments...
I20260926 02:07:39.708268 25687 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 02:07:39.713177 25244 tablet_copy_service.cc:342] P 16268d1d4f86406b9faa8950ac12a76d: Request end of tablet copy session 4d7e2176af0a475eab3956ac474f6320-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.159.189@KRBTEST.COM'} at 127.0.0.1:55588
I20260926 02:07:39.713459 25244 tablet_copy_service.cc:433] P 16268d1d4f86406b9faa8950ac12a76d: ending tablet copy session 4d7e2176af0a475eab3956ac474f6320-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4d7e2176af0a475eab3956ac474f6320
I20260926 02:07:39.724403 25687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:39.727289 25803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:39.731297 25804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:39.735751 25687 server_base.cc:1034] running on GCE node
W20260926 02:07:39.736140 25806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:39.736586 25687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:39.737248 25687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:39.738466 25687 hybrid_clock.cc:648] HybridClock initialized: now 1790388459738406 us; error 26 us; skew 500 ppm
I20260926 02:07:39.738721 25687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:39 dist-test-slave-03rn krb5kdc[25179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388459, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:39.740933 25687 init.cc:377] Logged in from keytab as oryx/127.18.159.189@KRBTEST.COM (short username oryx)
I20260926 02:07:39.748448 25687 webserver.cc:460] Webserver started at http://127.18.159.189:33817/ using document root <none> and password file <none>
I20260926 02:07:39.748859 25687 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:39.748977 25687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:39.751494 25687 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 02:07:39.753448 25813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:39.753870 25687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 02:07:39.754021 25687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/wal
uuid: "4d7e2176af0a475eab3956ac474f6320"
format_stamp: "Formatted at 2026-09-26 02:07:39 on dist-test-slave-03rn"
server_key: "a1551a08d10c87af79b68260efc3cd55"
server_key_iv: "3522c9796a1fe2ae667ef38a72c27e49"
server_key_version: "encryptionkey@0"
I20260926 02:07:39.754192 25687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:39.772987 25687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:39.776460 25687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:39.813942 25687 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.189:43111
I20260926 02:07:39.814020 25864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.189:43111 every 8 connection(s)
I20260926 02:07:39.817891 25687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-1/data/info.pb
I20260926 02:07:39.826939 25865 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:07:39.831512 25865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: Bootstrap starting.
I20260926 02:07:39.846623 25865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:07:39.847182 25865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: Bootstrap complete.
I20260926 02:07:39.848259 25865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: NON_VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: true } }
I20260926 02:07:39.848543 25865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4d7e2176af0a475eab3956ac474f6320, State: Initialized, Role: LEARNER
I20260926 02:07:39.848817 25865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: NON_VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: true } }
I20260926 02:07:39.849723 25868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: NON_VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: true } } }
I20260926 02:07:39.849983 25865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:07:39.850178 25868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: This master's current role is: LEARNER
I20260926 02:07:39.852377 25681 heartbeater.cc:503] Master 127.18.159.190:35357 was elected leader, sending a full tablet report...
I20260926 02:07:39.860590 25879 catalog_manager.cc:1259] Loaded cluster ID: 1fd63064f7554b2a889942cd3cb66892
I20260926 02:07:39.860705 25879 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: loading cluster ID for follower catalog manager: success
I20260926 02:07:39.864454 25879 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: acquiring CA information for follower catalog manager: success
I20260926 02:07:39.867868 25879 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 02:07:40.038941 25783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:40.039379 25783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:40.039601 25783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:40.049469 25783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:07:40.049700 25783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:40.049839 25783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:40.050035 25783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:07:40.050173 25783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:07:40.064314 25783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:35357,127.18.159.189:43111,127.18.159.188:40445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.188.keytab
--principal=oryx/127.18.159.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.188:40445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.159.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:40.065999 25783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:40.067639 25783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 02:07:40.083619 25785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: NON_VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: true } } }
I20260926 02:07:40.084031 25785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: This master's current role is: LEADER
W20260926 02:07:40.086967 25884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:40.087260 25885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:40.088138 25887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:40.089344 25783 server_base.cc:1034] running on GCE node
I20260926 02:07:40.090387 25783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:40.092296 25783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:40.094457 25889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.083774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:43111 (local address 127.0.0.1:37818)
0926 02:07:40.085587 (+ 1813us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:40.085609 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:40.085878 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:40.087012 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:40.087018 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:40.087050 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:40.087314 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:40.087330 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.089831 (+ 2501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.089837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:40.091230 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:40.091253 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.093317 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.093324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:40.093588 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.093658 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 02:07:40.093775 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 02:07:40.094002 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1603,"thread_start_us":1441,"threads_started":1}
I20260926 02:07:40.094463 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.084068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.18.159.189:43111)
0926 02:07:40.085136 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:40.085140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:40.086041 (+ 901us) server_negotiation.cc:407] Connection header received
0926 02:07:40.086133 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:40.086137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:40.086248 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:40.086556 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:40.087511 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.089645 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.091411 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.093072 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.093341 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.093538 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:40.093895 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 02:07:40.094042 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":891,"thread_start_us":658,"threads_started":1}
I20260926 02:07:40.094736 25783 hybrid_clock.cc:648] HybridClock initialized: now 1790388460094555 us; error 123 us; skew 500 ppm
I20260926 02:07:40.095275 25783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:07:40.095854 25868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16268d1d4f86406b9faa8950ac12a76d. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: NON_VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: true } } }
I20260926 02:07:40.096230 25868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: This master's current role is: LEARNER
I20260926 02:07:40.097096 25785 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d: attempting to promote NON_VOTER 4d7e2176af0a475eab3956ac474f6320 to VOTER
I20260926 02:07:40.098459 25785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } }
I20260926 02:07:40.100642 25840 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 1 LEARNER]: Refusing update from remote peer 16268d1d4f86406b9faa8950ac12a76d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 02:07:40.101483 25888 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
Sep 26 02:07:40 dist-test-slave-03rn krb5kdc[25179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388460, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:40.104916 25783 init.cc:377] Logged in from keytab as oryx/127.18.159.188@KRBTEST.COM (short username oryx)
I20260926 02:07:40.111145 25783 webserver.cc:460] Webserver started at http://127.18.159.188:40835/ using document root <none> and password file <none>
I20260926 02:07:40.111531 25785 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4d7e2176af0a475eab3956ac474f6320 (127.18.159.189) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } }
I20260926 02:07:40.112471 25783 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:40.112697 25783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:40.113112 25783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:40.115772 25888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } }
I20260926 02:07:40.116307 25888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:40.116744 25888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } }
I20260926 02:07:40.117112 25888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:40.116539 25840 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4d7e2176af0a475eab3956ac474f6320 (127.18.159.189) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } }
I20260926 02:07:40.118666 25783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/data/instance:
uuid: "04c3c25b95fe45f48ba2a2ab12765cf9"
format_stamp: "Formatted at 2026-09-26 02:07:40 on dist-test-slave-03rn"
server_key: "04dbc2efcb74e14c553a61ac9e2e7f26"
server_key_iv: "93158942bc54564d50e51a4aed4cda1a"
server_key_version: "encryptionkey@0"
I20260926 02:07:40.120024 25783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/wal/instance:
uuid: "04c3c25b95fe45f48ba2a2ab12765cf9"
format_stamp: "Formatted at 2026-09-26 02:07:40 on dist-test-slave-03rn"
server_key: "04dbc2efcb74e14c553a61ac9e2e7f26"
server_key_iv: "93158942bc54564d50e51a4aed4cda1a"
server_key_version: "encryptionkey@0"
I20260926 02:07:40.120939 25868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } }
I20260926 02:07:40.121232 25868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:07:40.127203 25783 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.006s
I20260926 02:07:40.131213 25898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:40.132858 25783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:07:40.133113 25783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/wal
uuid: "04c3c25b95fe45f48ba2a2ab12765cf9"
format_stamp: "Formatted at 2026-09-26 02:07:40 on dist-test-slave-03rn"
server_key: "04dbc2efcb74e14c553a61ac9e2e7f26"
server_key_iv: "93158942bc54564d50e51a4aed4cda1a"
server_key_version: "encryptionkey@0"
I20260926 02:07:40.133339 25783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:40.156385 25783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:40.161783 25783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:40.162129 25783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:40.183549 25783 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.188:40445
I20260926 02:07:40.183609 25949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.188:40445 every 8 connection(s)
I20260926 02:07:40.185211 25783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/data/info.pb
I20260926 02:07:40.190304 25950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:40.190446 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 25783
I20260926 02:07:40.200034 25950 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:07:40 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.159.188@KRBTEST.COM
Sep 26 02:07:40 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388460, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.188@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM
I20260926 02:07:40.216454 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.192910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58586 (local address 127.18.159.188:40445)
0926 02:07:40.194006 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:40.194019 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:40.194067 (+ 48us) server_negotiation.cc:407] Connection header received
0926 02:07:40.195711 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:40.195737 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:40.196252 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:40.196729 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:40.198402 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.199863 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.201323 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.202021 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.202224 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.207204 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:40.207214 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:40.207228 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:40.207280 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:40.211583 (+ 4303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:40.212505 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:40.212512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:40.212520 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:40.212757 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:40.213205 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:40.213211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:40.213215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:40.213899 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:40.214130 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:40.214727 (+ 597us) server_negotiation.cc:299] Negotiation successful
0926 02:07:40.215183 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":536,"threads_started":1}
I20260926 02:07:40.219856 25954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.202727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.0.0.1:55598)
0926 02:07:40.203761 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:40.203805 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:40.203989 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:40.204425 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:40.204433 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:40.205063 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:40.205308 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:40.205316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.206727 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.206734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:40.208395 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:40.208416 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.209364 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.209372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:40.209623 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.210246 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:40.210273 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:40.213470 (+ 3197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:40.216971 (+ 3501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:40.216980 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:40.216987 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:40.217609 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:40.217959 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:40.217963 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:40.217966 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:40.218126 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:40.218793 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:40.218798 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:40.218955 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 02:07:40.219129 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 02:07:40.219367 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":659,"threads_started":1}
I20260926 02:07:40.219861 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.202903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55598 (local address 127.18.159.190:35357)
0926 02:07:40.203452 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:40.203460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:40.203907 (+ 447us) server_negotiation.cc:407] Connection header received
0926 02:07:40.204127 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:40.204132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:40.204247 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:40.204363 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:40.205451 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.206595 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.208548 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.209113 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.209245 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.213690 (+ 4445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:40.213695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:40.213699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:40.213732 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:40.216840 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:40.217721 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:40.217725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:40.217727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:40.217848 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:40.218284 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:40.218288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:40.218291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:40.218659 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:40.218822 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:40.219250 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 02:07:40.219484 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":290,"threads_started":1}
I20260926 02:07:40.221288 25950 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:07:40 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388460, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.188@KRBTEST.COM for oryx/127.18.159.189@KRBTEST.COM
I20260926 02:07:40.234297 25954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.222075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:43111 (local address 127.0.0.1:37824)
0926 02:07:40.222260 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:40.222280 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:40.222389 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:40.222675 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:40.222678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:40.223013 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:40.223155 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:40.223163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.224229 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.224235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:40.225190 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:40.225201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.225911 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.225915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:40.226049 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.226649 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:40.226666 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:40.228790 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:40.231903 (+ 3113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:40.231907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:40.231910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:40.232396 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:40.232807 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:40.232810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:40.232812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:40.232962 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:40.233579 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:40.233582 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:40.233720 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 02:07:40.233825 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 02:07:40.234062 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 02:07:40.234543 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.222027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37824 (local address 127.18.159.189:43111)
0926 02:07:40.222197 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:40.222201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:40.222344 (+ 143us) server_negotiation.cc:407] Connection header received
0926 02:07:40.222491 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:40.222495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:40.222557 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:40.222638 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:40.223310 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.224048 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.225327 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.225757 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.225858 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.228992 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:40.228997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:40.229001 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:40.229035 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:40.231756 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:40.232547 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:40.232552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:40.232554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:40.232704 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:40.233108 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:40.233113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:40.233116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:40.233459 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:40.233578 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:40.233987 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 02:07:40.234211 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:07:40.235517 25950 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:07:40 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388460, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.188@KRBTEST.COM for oryx/127.18.159.188@KRBTEST.COM
I20260926 02:07:40.250891 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.236579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58602 (local address 127.18.159.188:40445)
0926 02:07:40.236734 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:40.236738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:40.236753 (+ 15us) server_negotiation.cc:407] Connection header received
0926 02:07:40.236869 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:40.236873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:40.236926 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:40.237001 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:40.237803 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.238604 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.240304 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.240720 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.240846 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.245285 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:40.245290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:40.245294 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:40.245336 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:40.248050 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:40.248765 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:40.248769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:40.248771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:40.248886 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:40.249258 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:40.249262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:40.249264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:40.249573 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:40.249765 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:40.250192 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 02:07:40.250539 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":52}
I20260926 02:07:40.250913 25954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.236344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:40445 (local address 127.0.0.1:58602)
0926 02:07:40.236609 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:40.236625 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:40.236707 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:40.237047 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:40.237051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:40.237378 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:40.237671 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:40.237679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.238781 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.238785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:40.239951 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:40.239963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.241630 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.241636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:40.241854 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.242751 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:40.242771 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:40.245069 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:40.248279 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:40.248284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:40.248286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:40.248656 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:40.248984 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:40.248988 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:40.248990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:40.249130 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:40.249748 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:40.249753 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:40.249940 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 02:07:40.250085 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 02:07:40.250387 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 02:07:40.256219 25950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9: Bootstrap starting.
I20260926 02:07:40.259636 25950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:40.260800 25950 log.cc:824] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:40.263700 25950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9: No bootstrap required, opened a new log
I20260926 02:07:40.271469 25950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } }
I20260926 02:07:40.271970 25950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:40.272094 25950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c3c25b95fe45f48ba2a2ab12765cf9, State: Initialized, Role: FOLLOWER
I20260926 02:07:40.272742 25950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } }
I20260926 02:07:40.273681 25957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } } }
I20260926 02:07:40.274061 25957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:07:40.274943 25950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 02:07:40.284827 25968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:07:40.284973 25968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:07:40.288833 25783 master_runner.cc:428] Detected that this master 04c3c25b95fe45f48ba2a2ab12765cf9 is joining an existing cluster
I20260926 02:07:40.289003 25783 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.159.188:40445
I20260926 02:07:40.310878 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.293665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55600 (local address 127.18.159.190:35357)
0926 02:07:40.293831 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:40.293837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:40.293980 (+ 143us) server_negotiation.cc:407] Connection header received
0926 02:07:40.294263 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:40.294269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:40.294371 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:40.294453 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:40.296498 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.297690 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.300119 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.300797 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.300900 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.303630 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:40.303637 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:40.303642 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:40.303684 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:40.307373 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:40.308252 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:40.308258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:40.308262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:40.308427 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:40.308988 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:40.308993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:40.308997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:40.309381 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:40.309505 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:40.310263 (+ 758us) server_negotiation.cc:299] Negotiation successful
0926 02:07:40.310513 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:07:40.312350 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.293101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.0.0.1:55600)
0926 02:07:40.293861 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:40.293880 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:40.294089 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:40.294680 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:40.294690 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:40.295743 (+ 1053us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:40.296300 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:40.296318 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.297919 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.297940 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:40.299947 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:40.299964 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.301222 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.301228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:40.301417 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.302186 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:40.302212 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:40.303419 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:40.307554 (+ 4135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:40.307561 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:40.307565 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:40.308075 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:40.308590 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:40.308595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:40.308598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:40.308797 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:40.309603 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:40.309607 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:40.309865 (+ 258us) client_negotiation.cc:769] Sending connection context
0926 02:07:40.311434 (+ 1569us) client_negotiation.cc:240] Negotiation successful
0926 02:07:40.311792 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":88,"thread_start_us":716,"threads_started":1}
I20260926 02:07:40.313072 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.293771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58612 (local address 127.18.159.188:40445)
0926 02:07:40.294251 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:40.294257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:40.296401 (+ 2144us) server_negotiation.cc:407] Connection header received
0926 02:07:40.296613 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:40.296617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:40.296678 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:40.296744 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:40.298152 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.299350 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.301711 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.302535 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.302673 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.305522 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:40.305528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:40.305532 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:40.305566 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:40.309612 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:40.310495 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:40.310502 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:40.310506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:40.310620 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:40.311078 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:40.311083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:40.311085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:40.311781 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:40.312009 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:40.312460 (+ 451us) server_negotiation.cc:299] Negotiation successful
0926 02:07:40.312672 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":352}
I20260926 02:07:40.313191 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.294051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37826 (local address 127.18.159.189:43111)
0926 02:07:40.294287 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:40.294293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:40.294832 (+ 539us) server_negotiation.cc:407] Connection header received
0926 02:07:40.295074 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:40.295079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:40.295157 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:40.295228 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:40.297353 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.298413 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.300998 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.301651 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.301752 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.305568 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:40.305576 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:40.305580 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:40.305623 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:40.309415 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:40.310439 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:40.310448 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:40.310453 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:40.310615 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:40.311091 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:40.311096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:40.311099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:40.311412 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:40.311525 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:40.312581 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0926 02:07:40.312813 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260926 02:07:40.314154 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.293343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:40445 (local address 127.0.0.1:58612)
0926 02:07:40.296310 (+ 2967us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:40.296328 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:40.296494 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:40.296917 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:40.296921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:40.297594 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:40.297979 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:40.297995 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.299592 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.299597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:40.301547 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:40.301561 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.302749 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.302755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:40.302964 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.303875 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:40.303892 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:40.305324 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:40.309761 (+ 4437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:40.309765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:40.309770 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:40.310391 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:40.310747 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:40.310752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:40.310755 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:40.310963 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:40.311957 (+ 994us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:40.311962 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:40.312130 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 02:07:40.313312 (+ 1182us) client_negotiation.cc:240] Negotiation successful
0926 02:07:40.313643 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2619,"mutex_wait_us":23,"spinlock_wait_cycles":112640,"thread_start_us":2449,"threads_started":1}
I20260926 02:07:40.314667 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.293314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:43111 (local address 127.0.0.1:37826)
0926 02:07:40.294724 (+ 1410us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:40.294741 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:40.294928 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:40.295447 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:40.295452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:40.296765 (+ 1313us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:40.297159 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:40.297172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.298613 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.298618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:40.300800 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:40.300814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.302041 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.302048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:40.303040 (+ 992us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.303812 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:40.303845 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:40.305306 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:40.309595 (+ 4289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:40.309599 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:40.309603 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:40.310283 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:40.310747 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:40.310752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:40.310755 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:40.310963 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:40.312068 (+ 1105us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:40.312073 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:40.312260 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 02:07:40.313334 (+ 1074us) client_negotiation.cc:240] Negotiation successful
0926 02:07:40.313794 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1258,"mutex_wait_us":482,"spinlock_wait_cycles":1408,"thread_start_us":1092,"threads_started":1}
I20260926 02:07:40.323601 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 25285
Sep 26 02:07:40 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388457, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.190@KRBTEST.COM for oryx/127.18.159.188@KRBTEST.COM
I20260926 02:07:40.334853 25889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.318667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:40445 (local address 127.0.0.1:58618)
0926 02:07:40.318852 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:40.318873 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:40.318998 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:40.319554 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:40.319561 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:40.320094 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:40.320326 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:40.320339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.321398 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.321406 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:40.322319 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:40.322333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.323377 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.323383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:40.323590 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.324349 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:40.324372 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:40.327382 (+ 3010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:40.331581 (+ 4199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:40.331587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:40.331591 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:40.332212 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:40.332881 (+ 669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:40.332887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:40.332893 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:40.333057 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:40.333685 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:40.333692 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:40.333920 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 02:07:40.334094 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 02:07:40.334416 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 02:07:40.335216 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.319096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58618 (local address 127.18.159.188:40445)
0926 02:07:40.319247 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:40.319252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:40.319265 (+ 13us) server_negotiation.cc:407] Connection header received
0926 02:07:40.319338 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:40.319342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:40.319417 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:40.319529 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:40.320528 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.321264 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.322466 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.323222 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.323350 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.327586 (+ 4236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:40.327592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:40.327597 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:40.327641 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:40.331426 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:40.332584 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:40.332590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:40.332594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:40.332743 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:40.333199 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:40.333204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:40.333208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:40.333543 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:40.333698 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:40.334254 (+ 556us) server_negotiation.cc:299] Negotiation successful
0926 02:07:40.334495 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Sep 26 02:07:40 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.159.129@KRBTEST.COM
I20260926 02:07:40.337191 25223 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.18.159.188:40445: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9"
member_type: NON_VOTER
last_known_addr {
host: "127.18.159.188"
port: 40445
}
attrs {
promote: true
}
}
dest_uuid: "16268d1d4f86406b9faa8950ac12a76d"
cas_config_opid_index: 6
I20260926 02:07:40.338088 25223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: NON_VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: true } }
I20260926 02:07:40.340562 25840 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 1 FOLLOWER]: Refusing update from remote peer 16268d1d4f86406b9faa8950ac12a76d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 02:07:40.341261 25785 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 02:07:40.341446 25925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:40.346060 25977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.324755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49806 (local address 127.18.159.129:34533)
0926 02:07:40.325454 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:40.325463 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:40.325493 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:07:40.325558 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:40.325567 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:40.325849 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:40.326075 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:40.327698 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.328993 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.330772 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.331620 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.332338 (+ 718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.337237 (+ 4899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:40.337250 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:40.337265 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:40.337327 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:40.341801 (+ 4474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:40.342811 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:40.342822 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:40.342827 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:40.343002 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:40.343532 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:40.343537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:40.343541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:40.344010 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:40.345134 (+ 1124us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:40.345325 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 02:07:40.345613 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":357,"threads_started":1}
I20260926 02:07:40.345872 25785 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 04c3c25b95fe45f48ba2a2ab12765cf9 (127.18.159.188) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: NON_VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: true } } }
I20260926 02:07:40.346371 25840 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 04c3c25b95fe45f48ba2a2ab12765cf9 (127.18.159.188) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: NON_VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: true } } }
I20260926 02:07:40.348030 25408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:40.348593 25925 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 1 FOLLOWER]: Refusing update from remote peer 16268d1d4f86406b9faa8950ac12a76d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 02:07:40.349761 25888 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: NON_VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:40.351169 25868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: NON_VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: true } } }
I20260926 02:07:40.351544 25868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:07:40.360147 25925 raft_consensus.cc:858] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: NON_VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } }
W20260926 02:07:40.360394 25925 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: NON_VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } }
I20260926 02:07:40.362936 25785 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.18.159.188:40445
I20260926 02:07:40.363744 25783 master.cc:537] Master@127.18.159.188:40445 shutting down...
I20260926 02:07:40.364533 25785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: NON_VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: true } } }
I20260926 02:07:40.364518 25888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: NON_VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: true } } }
I20260926 02:07:40.364981 25785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:40.365031 25888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:40.400511 25925 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4d7e2176af0a475eab3956ac474f6320 (127.18.159.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: NON_VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: true } } }
I20260926 02:07:40.405046 25985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: NON_VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: true } } }
I20260926 02:07:40.405378 25985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 02:07:40.405376 25957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16268d1d4f86406b9faa8950ac12a76d. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: NON_VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: true } } }
I20260926 02:07:40.405634 25957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 02:07:40.405941 25982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: NON_VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: true } } }
I20260926 02:07:40.406253 25982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 02:07:40.408784 25979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: NON_VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: true } } }
I20260926 02:07:40.409157 25979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: This master's current role is: LEADER
W20260926 02:07:40.409684 25211 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 02:07:40.410698 25783 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260926 02:07:40.411194 25783 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260926 02:07:40.411403 25783 tablet_replica.cc:330] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9: stopping tablet replica
W20260926 02:07:40.411870 25211 consensus_peers.cc:487] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d -> Peer 04c3c25b95fe45f48ba2a2ab12765cf9 (127.18.159.188:40445): Couldn't send request to peer 04c3c25b95fe45f48ba2a2ab12765cf9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 02:07:40.429661 25783 master.cc:552] Master@127.18.159.188:40445 shutdown complete.
I20260926 02:07:40.429764 25783 master_runner.cc:305] Clearing existing system tablet
I20260926 02:07:40.430173 25783 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:40.430289 25783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:40.433454 25783 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260926 02:07:40.436534 25988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:40.437052 25783 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:40.437311 25783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:07:40.437417 25783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/wal
uuid: "04c3c25b95fe45f48ba2a2ab12765cf9"
format_stamp: "Formatted at 2026-09-26 02:07:40 on dist-test-slave-03rn"
server_key: "04dbc2efcb74e14c553a61ac9e2e7f26"
server_key_iv: "93158942bc54564d50e51a4aed4cda1a"
server_key_version: "encryptionkey@0"
I20260926 02:07:40.442162 25783 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:40.449966 25783 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 02:07:40.450158 25783 log.cc:1196] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 02:07:40.450707 25783 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9: Deleting consensus metadata
I20260926 02:07:40.451041 25783 master_runner.cc:315] Copying system tablet from 127.18.159.190:35357
I20260926 02:07:40.461211 25783 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9: tablet copy: Beginning tablet copy session from remote peer at address 127.18.159.190:35357
I20260926 02:07:40.465816 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 25285
I20260926 02:07:40.475745 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.462422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55608 (local address 127.18.159.190:35357)
0926 02:07:40.462619 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:40.462628 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:40.462909 (+ 281us) server_negotiation.cc:407] Connection header received
0926 02:07:40.463201 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:40.463208 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:40.463387 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:40.463480 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:40.464816 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.465806 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.467334 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.467789 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.467937 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.469977 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:40.469983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:40.469989 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:40.470030 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:40.472789 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:40.473477 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:40.473482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:40.473484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:40.473680 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:40.474087 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:40.474093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:40.474096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:40.474375 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:40.474591 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:40.475061 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 02:07:40.475307 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:07:40.475739 25994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.462034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.0.0.1:55608)
0926 02:07:40.462793 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:40.462818 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:40.463030 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:40.463710 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:40.463715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:40.464357 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:40.464611 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:40.464628 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.465988 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.465993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:40.467165 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:40.467179 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.467970 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.467976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:40.468232 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.468933 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:40.468952 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:40.469748 (+ 796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:40.472947 (+ 3199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:40.472953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:40.472958 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:40.473327 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:40.473795 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:40.473801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:40.473805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:40.473973 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:40.474511 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:40.474516 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:40.474764 (+ 248us) client_negotiation.cc:769] Sending connection context
0926 02:07:40.474931 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 02:07:40.475279 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":438,"threads_started":1}
I20260926 02:07:40.476826 25244 tablet_copy_service.cc:140] P 16268d1d4f86406b9faa8950ac12a76d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 04c3c25b95fe45f48ba2a2ab12765cf9 ({username='oryx', principal='oryx/127.18.159.188@KRBTEST.COM'} at 127.0.0.1:55608)
I20260926 02:07:40.477005 25244 tablet_copy_service.cc:161] P 16268d1d4f86406b9faa8950ac12a76d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 04c3c25b95fe45f48ba2a2ab12765cf9 at {username='oryx', principal='oryx/127.18.159.188@KRBTEST.COM'} at 127.0.0.1:55608: session id = 04c3c25b95fe45f48ba2a2ab12765cf9-00000000000000000000000000000000
I20260926 02:07:40.479526 25244 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 02:07:40.481559 25783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 02:07:40.488574 25783 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9: tablet copy: Starting download of 0 data blocks...
I20260926 02:07:40.488890 25783 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9: tablet copy: Starting download of 1 WAL segments...
I20260926 02:07:40.491647 25783 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 02:07:40.491683 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:34533
--local_ip_for_outbound_sockets=127.18.159.129
--tserver_master_addrs=127.18.159.190:35357
--webserver_port=46123
--webserver_interface=127.18.159.129
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.129.keytab
--principal=oryx/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.159.190:35357,127.18.159.189:43111
--tserver_master_addrs=127.18.159.190:35357,127.18.159.189:43111,127.18.159.188:40445 with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:07:40.496973 25244 tablet_copy_service.cc:342] P 16268d1d4f86406b9faa8950ac12a76d: Request end of tablet copy session 04c3c25b95fe45f48ba2a2ab12765cf9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.159.188@KRBTEST.COM'} at 127.0.0.1:55608
I20260926 02:07:40.497134 25244 tablet_copy_service.cc:433] P 16268d1d4f86406b9faa8950ac12a76d: ending tablet copy session 04c3c25b95fe45f48ba2a2ab12765cf9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 04c3c25b95fe45f48ba2a2ab12765cf9
I20260926 02:07:40.510035 25783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:40.513808 25996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:40.514839 25997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:40.526592 25999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:40.533644 25783 server_base.cc:1034] running on GCE node
I20260926 02:07:40.534082 25783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:40.534739 25783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:40.535965 25783 hybrid_clock.cc:648] HybridClock initialized: now 1790388460535918 us; error 27 us; skew 500 ppm
I20260926 02:07:40.536243 25783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:40 dist-test-slave-03rn krb5kdc[25179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388460, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:40.538550 25783 init.cc:377] Logged in from keytab as oryx/127.18.159.188@KRBTEST.COM (short username oryx)
I20260926 02:07:40.544189 25783 webserver.cc:460] Webserver started at http://127.18.159.188:40085/ using document root <none> and password file <none>
I20260926 02:07:40.544587 25783 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:40.544692 25783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:40.547966 25783 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 02:07:40.549942 26006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:40.550491 25783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260926 02:07:40.550617 25783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/wal
uuid: "04c3c25b95fe45f48ba2a2ab12765cf9"
format_stamp: "Formatted at 2026-09-26 02:07:40 on dist-test-slave-03rn"
server_key: "04dbc2efcb74e14c553a61ac9e2e7f26"
server_key_iv: "93158942bc54564d50e51a4aed4cda1a"
server_key_version: "encryptionkey@0"
I20260926 02:07:40.550800 25783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:40.582446 25783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:40.587023 25783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:40.619961 25783 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.188:40445
I20260926 02:07:40.620007 26057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.188:40445 every 8 connection(s)
I20260926 02:07:40.621160 25783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/master-2/data/info.pb
I20260926 02:07:40.626929 26058 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:07:40.631049 26058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9: Bootstrap starting.
I20260926 02:07:40.647392 26058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:07:40.647935 26058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9: Bootstrap complete.
I20260926 02:07:40.649092 26058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: NON_VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: true } }
I20260926 02:07:40.649412 26058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 04c3c25b95fe45f48ba2a2ab12765cf9, State: Initialized, Role: LEARNER
I20260926 02:07:40.649686 26058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: NON_VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: true } }
I20260926 02:07:40.650651 26061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: NON_VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: true } } }
I20260926 02:07:40.651024 26058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:07:40.651199 26061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [sys.catalog]: This master's current role is: LEARNER
I20260926 02:07:40.666549 26072 catalog_manager.cc:1259] Loaded cluster ID: 1fd63064f7554b2a889942cd3cb66892
I20260926 02:07:40.666672 26072 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9: loading cluster ID for follower catalog manager: success
I20260926 02:07:40.670168 26072 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9: acquiring CA information for follower catalog manager: success
I20260926 02:07:40.673436 26072 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:07:40.849347 25888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: NON_VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: true } } }
I20260926 02:07:40.849680 25888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:40.856930 26073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.849645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:40445 (local address 127.0.0.1:58620)
0926 02:07:40.850338 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:40.850361 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:40.850520 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:40.850973 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:40.850977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:40.851018 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:40.851376 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:40.851387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.853094 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.853101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:40.854341 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:40.854355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.855966 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.855972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:40.856378 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.856412 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 02:07:40.856511 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 02:07:40.856628 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":456,"threads_started":1}
I20260926 02:07:40.857074 26074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:40.849848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58620 (local address 127.18.159.188:40445)
0926 02:07:40.850670 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:40.850675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:40.850690 (+ 15us) server_negotiation.cc:407] Connection header received
0926 02:07:40.850750 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:40.850754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:40.850821 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:40.850955 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:40.851568 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.852953 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.854531 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:40.855731 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:40.855944 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:40.856160 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:40.856562 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 02:07:40.856719 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":536,"threads_started":1}
I20260926 02:07:40.857903 26033 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260926 02:07:40.858376 26061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16268d1d4f86406b9faa8950ac12a76d. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: NON_VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: true } } }
I20260926 02:07:40.858716 26061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [sys.catalog]: This master's current role is: LEARNER
I20260926 02:07:40.859083 25979 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d: attempting to promote NON_VOTER 04c3c25b95fe45f48ba2a2ab12765cf9 to VOTER
I20260926 02:07:40.859755 25979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: false } }
I20260926 02:07:40.861714 25840 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 1 FOLLOWER]: Refusing update from remote peer 16268d1d4f86406b9faa8950ac12a76d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 02:07:40.862010 26033 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 1 LEARNER]: Refusing update from remote peer 16268d1d4f86406b9faa8950ac12a76d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 02:07:40.862322 25979 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 02:07:40.862855 25979 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 02:07:40.864969 25888 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 04c3c25b95fe45f48ba2a2ab12765cf9 (127.18.159.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: false } } }
I20260926 02:07:40.865610 25840 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 04c3c25b95fe45f48ba2a2ab12765cf9 (127.18.159.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: false } } }
I20260926 02:07:40.868827 26076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: false } } }
I20260926 02:07:40.869202 26076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:07:40.870887 25888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: false } } }
I20260926 02:07:40.871187 25888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:40.870993 25785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: false } } }
I20260926 02:07:40.871403 25785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:40.870986 26033 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 04c3c25b95fe45f48ba2a2ab12765cf9 (127.18.159.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: false } } }
I20260926 02:07:40.871533 25888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: false } } }
I20260926 02:07:40.871762 25888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:40.874228 26061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16268d1d4f86406b9faa8950ac12a76d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: false } } }
I20260926 02:07:40.874537 26061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 02:07:40.906672 25995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:40.906991 25995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:40.907166 25995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:40.914738 25995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:40.914873 25995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:40.915072 25995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:07:40.925683 25995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.129.keytab
--principal=oryx/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:34533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=46123
--tserver_master_addrs=127.18.159.190:35357,127.18.159.189:43111,127.18.159.188:40445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:40.927052 25995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:40.928506 25995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:40.942464 26082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:40.942442 26085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:40.942531 26083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:40.949251 25995 server_base.cc:1034] running on GCE node
I20260926 02:07:40.950032 25995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:40.951578 25995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:40.952836 25995 hybrid_clock.cc:648] HybridClock initialized: now 1790388460952795 us; error 43 us; skew 500 ppm
I20260926 02:07:40.953194 25995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:40 dist-test-slave-03rn krb5kdc[25179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388460, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:40.957062 25995 init.cc:377] Logged in from keytab as oryx/127.18.159.129@KRBTEST.COM (short username oryx)
I20260926 02:07:40.959946 25995 webserver.cc:460] Webserver started at http://127.18.159.129:46123/ using document root <none> and password file <none>
I20260926 02:07:40.961678 25995 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:40.961860 25995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:40.968871 25995 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:07:40.971714 26092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:40.973294 25995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:07:40.973500 25995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "3f30c24fe8b7457ba6f65e5ef22fee66"
format_stamp: "Formatted at 2026-09-26 02:07:37 on dist-test-slave-03rn"
server_key: "338e66e6369f9413d9bc8cead76fcf1d"
server_key_iv: "399888b8df1bccb781147b21dc5a9b49"
server_key_version: "encryptionkey@0"
I20260926 02:07:40.974620 25995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:40.999428 25995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:41.004482 25995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:41.004779 25995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:41.005851 25995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:41.007268 25995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:41.007360 25995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:41.007433 25995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:41.007489 25995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:41.045964 25995 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:34533
I20260926 02:07:41.046051 26204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:34533 every 8 connection(s)
I20260926 02:07:41.047410 25995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:07:41.056444 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 25995
I20260926 02:07:41.056859 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 25419
Sep 26 02:07:41 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388460, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.188@KRBTEST.COM
Sep 26 02:07:41 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388460, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM
Sep 26 02:07:41 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388460, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.189@KRBTEST.COM
Sep 26 02:07:41 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.159.130@KRBTEST.COM
I20260926 02:07:41.083946 26074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:41.051048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:35647 (local address 127.18.159.188:40445)
0926 02:07:41.051250 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:41.051259 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:41.052563 (+ 1304us) server_negotiation.cc:407] Connection header received
0926 02:07:41.056204 (+ 3641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:41.056214 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:41.056329 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:41.056425 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:41.061041 (+ 4616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.062561 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.066593 (+ 4032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.067515 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.067654 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:41.074242 (+ 6588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:41.074251 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:41.074255 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:41.074303 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:41.079221 (+ 4918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:41.080535 (+ 1314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:41.080543 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:41.080548 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:41.080712 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:41.081262 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:41.081270 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:41.081273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:41.081697 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:41.081832 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:41.083253 (+ 1421us) server_negotiation.cc:299] Negotiation successful
0926 02:07:41.083555 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 02:07:41.086652 26210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:41.051200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:40445 (local address 127.18.159.129:35647)
0926 02:07:41.052280 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:41.052341 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:41.055785 (+ 3444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:41.057110 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:41.057128 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:41.058087 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:41.060792 (+ 2705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:41.060827 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.062856 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.062866 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:41.066327 (+ 3461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:41.066354 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.068113 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.068121 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:41.068539 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:41.069524 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:41.069566 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:41.073864 (+ 4298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:41.079415 (+ 5551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:41.079425 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:41.079447 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:41.080359 (+ 912us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:41.080860 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:41.080865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:41.080869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:41.081118 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:41.081936 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:41.081945 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:41.082806 (+ 861us) client_negotiation.cc:769] Sending connection context
0926 02:07:41.084308 (+ 1502us) client_negotiation.cc:240] Negotiation successful
0926 02:07:41.084945 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":1601,"threads_started":1}
I20260926 02:07:41.088649 26205 heartbeater.cc:347] Connected to a master server at 127.18.159.188:40445
I20260926 02:07:41.089011 26205 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:41.089358 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:41.052073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.129:51095)
0926 02:07:41.058838 (+ 6765us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:41.058865 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:41.059022 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:41.067971 (+ 8949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:41.067976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:41.068833 (+ 857us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:41.069147 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:41.069157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.072258 (+ 3101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.072263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:41.074457 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:41.074470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.075582 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.075591 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:41.075818 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:41.076467 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:41.076489 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:41.080246 (+ 3757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:41.085547 (+ 5301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:41.085554 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:41.085558 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:41.086225 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:41.087082 (+ 857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:41.087088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:41.087092 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:41.087259 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:41.088024 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:41.088031 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:41.088245 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 02:07:41.088406 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 02:07:41.088728 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6575,"thread_start_us":10613,"threads_started":1}
I20260926 02:07:41.089738 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:41.051200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:43111 (local address 127.18.159.129:35447)
0926 02:07:41.063101 (+ 11901us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:41.063127 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:41.063283 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:41.072194 (+ 8911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:41.072202 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:41.073570 (+ 1368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:41.073931 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:41.073944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.075496 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.075501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:41.077175 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:41.077191 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.078322 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.078328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:41.078519 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:41.079113 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:41.079135 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:41.082456 (+ 3321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:41.086682 (+ 4226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:41.086690 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:41.086695 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:41.087329 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:41.087778 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:41.087783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:41.087786 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:41.088001 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:41.088646 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:41.088654 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:41.088797 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 02:07:41.088927 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 02:07:41.089235 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11690,"mutex_wait_us":1113,"spinlock_wait_cycles":22272,"thread_start_us":6092,"threads_started":1}
I20260926 02:07:41.090076 26205 heartbeater.cc:511] Master 127.18.159.188:40445 requested a full tablet report, sending...
I20260926 02:07:41.090534 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:41.052099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:51095 (local address 127.18.159.190:35357)
0926 02:07:41.067406 (+ 15307us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:41.067415 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:41.067443 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:07:41.067521 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:41.067530 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:41.067634 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:41.070701 (+ 3067us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:41.070862 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.072122 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.074674 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.075347 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.076192 (+ 845us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:41.080574 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:41.080585 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:41.080592 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:41.080639 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:41.085350 (+ 4711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:41.086455 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:41.086461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:41.086464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:41.086631 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:41.087440 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:41.087446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:41.087452 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:41.087889 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:41.089173 (+ 1284us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:41.089326 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 02:07:41.090133 (+ 807us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15125,"thread_start_us":15668,"threads_started":1}
I20260926 02:07:41.091162 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:41.050691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:35447 (local address 127.18.159.189:43111)
0926 02:07:41.071827 (+ 21136us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:41.071833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:41.071855 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:07:41.071932 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:41.071937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:41.072012 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:41.072135 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:41.074132 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.075304 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.077404 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.078117 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.078274 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:41.082707 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:41.082714 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:41.082719 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:41.082763 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:41.086513 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:41.087493 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:41.087501 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:41.087505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:41.087644 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:41.088150 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:41.088179 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:41.088182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:41.088513 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:41.090117 (+ 1604us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:41.090255 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 02:07:41.090483 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20962,"thread_start_us":21565,"threads_started":1}
I20260926 02:07:41.091405 26206 heartbeater.cc:347] Connected to a master server at 127.18.159.190:35357
I20260926 02:07:41.091523 26206 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:41.092064 26206 heartbeater.cc:511] Master 127.18.159.190:35357 requested a full tablet report, sending...
I20260926 02:07:41.093030 26207 heartbeater.cc:347] Connected to a master server at 127.18.159.189:43111
I20260926 02:07:41.093161 26207 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:41.093338 26023 ts_manager.cc:194] Registered new tserver with Master: 3f30c24fe8b7457ba6f65e5ef22fee66 (127.18.159.129:34533)
I20260926 02:07:41.094429 25223 ts_manager.cc:194] Re-registered known tserver with Master: 3f30c24fe8b7457ba6f65e5ef22fee66 (127.18.159.129:34533)
I20260926 02:07:41.094527 26215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:41.058725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57102 (local address 127.18.159.130:42645)
0926 02:07:41.068192 (+ 9467us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:41.068202 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:41.068232 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:07:41.068308 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:41.068319 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:41.068602 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:41.068797 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:41.071003 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.073173 (+ 2170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.075502 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.076667 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.077488 (+ 821us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:41.083135 (+ 5647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:41.083155 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:41.083173 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:41.083215 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:41.087038 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:41.088067 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:41.088076 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:41.088081 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:41.088285 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:41.091226 (+ 2941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:41.091234 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:41.091239 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:41.092568 (+ 1329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:41.092872 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:41.093832 (+ 960us) server_negotiation.cc:299] Negotiation successful
0926 02:07:41.094128 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9249,"thread_start_us":9074,"threads_started":1}
I20260926 02:07:41.095310 25223 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:51095
I20260926 02:07:41.095950 26207 heartbeater.cc:511] Master 127.18.159.189:43111 requested a full tablet report, sending...
I20260926 02:07:41.097867 25830 ts_manager.cc:194] Registered new tserver with Master: 3f30c24fe8b7457ba6f65e5ef22fee66 (127.18.159.129:34533)
I20260926 02:07:41.098060 25542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:41.181819 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 25419
I20260926 02:07:41.198274 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:42645
--local_ip_for_outbound_sockets=127.18.159.130
--tserver_master_addrs=127.18.159.190:35357
--webserver_port=40743
--webserver_interface=127.18.159.130
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.130.keytab
--principal=oryx/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.159.190:35357,127.18.159.189:43111
--tserver_master_addrs=127.18.159.190:35357,127.18.159.189:43111,127.18.159.188:40445 with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:41.563360 26217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:41.563705 26217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:41.563880 26217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:41.571815 26217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:41.572001 26217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:41.572203 26217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:07:41.582681 26217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.130.keytab
--principal=oryx/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:42645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=40743
--tserver_master_addrs=127.18.159.190:35357,127.18.159.189:43111,127.18.159.188:40445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:41.584106 26217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:41.585575 26217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:41.600826 26222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:41.600723 26225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:41.600768 26223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:41.601806 26217 server_base.cc:1034] running on GCE node
I20260926 02:07:41.602510 26217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:41.603529 26217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:41.604765 26217 hybrid_clock.cc:648] HybridClock initialized: now 1790388461604697 us; error 68 us; skew 500 ppm
I20260926 02:07:41.605123 26217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:41 dist-test-slave-03rn krb5kdc[25179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388461, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:41.608613 26217 init.cc:377] Logged in from keytab as oryx/127.18.159.130@KRBTEST.COM (short username oryx)
I20260926 02:07:41.610265 26217 webserver.cc:460] Webserver started at http://127.18.159.130:40743/ using document root <none> and password file <none>
I20260926 02:07:41.611009 26217 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:41.611181 26217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:41.617237 26217 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 02:07:41.620482 26232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:41.622085 26217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 02:07:41.622277 26217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "17e22f81cc54435fa3db75d74f44e284"
format_stamp: "Formatted at 2026-09-26 02:07:38 on dist-test-slave-03rn"
server_key: "2c60bc3b2c151ae6045c866f9a434fc4"
server_key_iv: "69f0cd51b505f08d95d83e92e7a95f14"
server_key_version: "encryptionkey@0"
I20260926 02:07:41.623435 26217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:41.642668 26217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:41.647544 26217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:41.647929 26217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:41.649061 26217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:41.650635 26217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:41.650725 26217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:41.650843 26217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:41.650928 26217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:41.689446 26217 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:42645
I20260926 02:07:41.689574 26344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:42645 every 8 connection(s)
I20260926 02:07:41.691037 26217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:07:41.701455 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 26217
I20260926 02:07:41.701802 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 25553
Sep 26 02:07:41 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.159.131@KRBTEST.COM
Sep 26 02:07:41 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388461, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM
Sep 26 02:07:41 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388461, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.188@KRBTEST.COM
Sep 26 02:07:41 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388461, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.189@KRBTEST.COM
I20260926 02:07:41.742322 26351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:41.695571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:47629 (local address 127.18.159.190:35357)
0926 02:07:41.710730 (+ 15159us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:41.710734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:41.713646 (+ 2912us) server_negotiation.cc:407] Connection header received
0926 02:07:41.720116 (+ 6470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:41.720122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:41.720248 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:41.720323 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:41.724420 (+ 4097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.725561 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.727480 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.728125 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.728255 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:41.733427 (+ 5172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:41.733433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:41.733436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:41.733502 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:41.738001 (+ 4499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:41.739145 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:41.739151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:41.739154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:41.739357 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:41.739868 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:41.739874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:41.739877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:41.740283 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:41.740427 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:41.741712 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0926 02:07:41.741954 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15050,"thread_start_us":15147,"threads_started":1}
I20260926 02:07:41.742527 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:41.695675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:45547 (local address 127.18.159.189:43111)
0926 02:07:41.719987 (+ 24312us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:41.719996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:41.720017 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:07:41.720076 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:41.720080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:41.720180 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:41.720284 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:41.722772 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.723998 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.726972 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.727857 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.727990 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:41.735015 (+ 7025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:41.735024 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:41.735029 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:41.735077 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:41.739118 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:41.740059 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:41.740065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:41.740068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:41.740249 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:41.740692 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:41.740697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:41.740700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:41.741072 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:41.741189 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:41.741906 (+ 717us) server_negotiation.cc:299] Negotiation successful
0926 02:07:41.742132 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24190,"thread_start_us":24097,"threads_started":1}
I20260926 02:07:41.743252 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:41.703026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47830 (local address 127.18.159.131:41509)
0926 02:07:41.716785 (+ 13759us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:41.716799 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:41.716840 (+ 41us) server_negotiation.cc:407] Connection header received
0926 02:07:41.716926 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:41.716940 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:41.717270 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:41.717502 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:41.720415 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.722074 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.724222 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.725415 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.725563 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:41.733394 (+ 7831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:41.733413 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:41.733434 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:41.733498 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:41.738297 (+ 4799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:41.739412 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:41.739423 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:41.739430 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:41.739597 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:41.740079 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:41.740083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:41.740086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:41.740552 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:41.740843 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:41.742089 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0926 02:07:41.742347 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13558,"thread_start_us":13397,"threads_started":1}
I20260926 02:07:41.743764 26350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:41.695330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.130:47629)
0926 02:07:41.713462 (+ 18132us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:41.713511 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:41.717193 (+ 3682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:41.720533 (+ 3340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:41.720537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:41.721225 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:41.724203 (+ 2978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:41.724222 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.725758 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.725763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:41.727302 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:41.727315 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.728542 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.728547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:41.728741 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:41.729599 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:41.729624 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:41.732921 (+ 3297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:41.738549 (+ 5628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:41.738552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:41.738554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:41.738993 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:41.739518 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:41.739521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:41.739524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:41.739732 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:41.740777 (+ 1045us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:41.740781 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:41.741269 (+ 488us) client_negotiation.cc:769] Sending connection context
0926 02:07:41.741604 (+ 335us) client_negotiation.cc:240] Negotiation successful
0926 02:07:41.742168 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17799,"mutex_wait_us":2131,"spinlock_wait_cycles":43392,"thread_start_us":22475,"threads_started":1}
I20260926 02:07:41.743918 26353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:41.695330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:43111 (local address 127.18.159.130:45547)
0926 02:07:41.719374 (+ 24044us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:41.719399 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:41.719541 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:41.720504 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:41.720509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:41.721226 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:41.722528 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:41.722553 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.724200 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.724206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:41.726780 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:41.726799 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.728388 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.728396 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:41.728640 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:41.729802 (+ 1162us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:41.729815 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:41.734746 (+ 4931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:41.739312 (+ 4566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:41.739317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:41.739321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:41.739902 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:41.740400 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:41.740404 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:41.740408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:41.740568 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:41.741286 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:41.741290 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:41.741460 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 02:07:41.741619 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 02:07:41.743098 (+ 1479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23841,"mutex_wait_us":1114,"spinlock_wait_cycles":17408,"thread_start_us":24906,"threads_started":1}
I20260926 02:07:41.744539 26349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:41.694449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:38705 (local address 127.18.159.188:40445)
0926 02:07:41.709892 (+ 15443us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:41.709903 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:41.718423 (+ 8520us) server_negotiation.cc:407] Connection header received
0926 02:07:41.718760 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:41.718766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:41.718872 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:41.719024 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:41.722746 (+ 3722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.723808 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.727002 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.727846 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.727979 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:41.733632 (+ 5653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:41.733642 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:41.733647 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:41.733689 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:41.737381 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:41.739174 (+ 1793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:41.739181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:41.739184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:41.739357 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:41.740200 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:41.740205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:41.740209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:41.740558 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:41.740715 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:41.743854 (+ 3139us) server_negotiation.cc:299] Negotiation successful
0926 02:07:41.744111 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15208,"thread_start_us":15484,"threads_started":1}
I20260926 02:07:41.743793 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:41.695436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:40445 (local address 127.18.159.130:38705)
0926 02:07:41.718272 (+ 22836us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:41.718313 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:41.718497 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:41.719604 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:41.719620 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:41.721231 (+ 1611us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:41.722526 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:41.722550 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.724010 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.724017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:41.726770 (+ 2753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:41.726785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:41.728232 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:41.728240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:41.728592 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:41.729556 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:41.729602 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:41.733387 (+ 3785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:41.737660 (+ 4273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:41.737674 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:41.737695 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:41.738356 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:41.739890 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:41.739894 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:41.739897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:41.740043 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:41.740689 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:41.740697 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:41.741269 (+ 572us) client_negotiation.cc:769] Sending connection context
0926 02:07:41.742622 (+ 1353us) client_negotiation.cc:240] Negotiation successful
0926 02:07:41.742942 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22690,"mutex_wait_us":637,"spinlock_wait_cycles":43776,"thread_start_us":17578,"threads_started":1}
I20260926 02:07:41.745784 26347 heartbeater.cc:347] Connected to a master server at 127.18.159.189:43111
I20260926 02:07:41.745759 26346 heartbeater.cc:347] Connected to a master server at 127.18.159.190:35357
I20260926 02:07:41.746131 25676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:41.746158 26347 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:41.746160 26346 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:41.746510 26345 heartbeater.cc:347] Connected to a master server at 127.18.159.188:40445
I20260926 02:07:41.746743 26345 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:41.747349 26346 heartbeater.cc:511] Master 127.18.159.190:35357 requested a full tablet report, sending...
I20260926 02:07:41.747375 26347 heartbeater.cc:511] Master 127.18.159.189:43111 requested a full tablet report, sending...
I20260926 02:07:41.747430 26345 heartbeater.cc:511] Master 127.18.159.188:40445 requested a full tablet report, sending...
I20260926 02:07:41.749753 25830 ts_manager.cc:194] Registered new tserver with Master: 17e22f81cc54435fa3db75d74f44e284 (127.18.159.130:42645)
I20260926 02:07:41.750639 25223 ts_manager.cc:194] Re-registered known tserver with Master: 17e22f81cc54435fa3db75d74f44e284 (127.18.159.130:42645)
I20260926 02:07:41.752848 25223 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:47629
I20260926 02:07:41.754230 26023 ts_manager.cc:194] Registered new tserver with Master: 17e22f81cc54435fa3db75d74f44e284 (127.18.159.130:42645)
I20260926 02:07:41.840250 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 25553
I20260926 02:07:41.857635 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:41509
--local_ip_for_outbound_sockets=127.18.159.131
--tserver_master_addrs=127.18.159.190:35357
--webserver_port=38061
--webserver_interface=127.18.159.131
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.131.keytab
--principal=oryx/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.159.190:35357,127.18.159.189:43111
--tserver_master_addrs=127.18.159.190:35357,127.18.159.189:43111,127.18.159.188:40445 with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:07:42.098224 26206 heartbeater.cc:503] Master 127.18.159.190:35357 was elected leader, sending a full tablet report...
W20260926 02:07:42.210055 26358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:42.210353 26358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:42.210507 26358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:42.218021 26358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:42.218164 26358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:42.218362 26358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:07:42.228686 26358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.131.keytab
--principal=oryx/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:41509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=38061
--tserver_master_addrs=127.18.159.190:35357,127.18.159.189:43111,127.18.159.188:40445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:42.230080 26358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:42.231439 26358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:42.246016 26363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:42.245906 26366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:42.245882 26364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:42.246970 26358 server_base.cc:1034] running on GCE node
I20260926 02:07:42.247640 26358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:42.248750 26358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:42.250017 26358 hybrid_clock.cc:648] HybridClock initialized: now 1790388462249966 us; error 49 us; skew 500 ppm
I20260926 02:07:42.250417 26358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:42 dist-test-slave-03rn krb5kdc[25179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388462, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:42.254565 26358 init.cc:377] Logged in from keytab as oryx/127.18.159.131@KRBTEST.COM (short username oryx)
I20260926 02:07:42.256542 26358 webserver.cc:460] Webserver started at http://127.18.159.131:38061/ using document root <none> and password file <none>
I20260926 02:07:42.257320 26358 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:42.257489 26358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:42.264043 26358 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 02:07:42.267439 26373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:42.269145 26358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:07:42.269392 26358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c"
format_stamp: "Formatted at 2026-09-26 02:07:38 on dist-test-slave-03rn"
server_key: "e8b468411bd4dbc8a0529a876a9ea6b4"
server_key_iv: "16709f93faccc67d7db705f248f908d7"
server_key_version: "encryptionkey@0"
I20260926 02:07:42.270478 26358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:42.295992 26358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:42.300570 26358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:42.300889 26358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:42.302037 26358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:42.303560 26358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:42.303650 26358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:42.303776 26358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:42.303889 26358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:42.339946 26358 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:41509
I20260926 02:07:42.340039 26485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:41509 every 8 connection(s)
I20260926 02:07:42.341528 26358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:07:42.351624 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 26358
Sep 26 02:07:42 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388462, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.189@KRBTEST.COM
I20260926 02:07:42.389024 26490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.344596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:54089 (local address 127.18.159.189:43111)
0926 02:07:42.364414 (+ 19818us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:42.364424 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:42.364453 (+ 29us) server_negotiation.cc:407] Connection header received
0926 02:07:42.364528 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:42.364536 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:42.364640 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:42.364770 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:42.368189 (+ 3419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.370010 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.372436 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.373115 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.373268 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.378759 (+ 5491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:42.378767 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:42.378772 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:42.378824 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:42.383682 (+ 4858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:42.385127 (+ 1445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:42.385138 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:42.385145 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:42.385345 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:42.386031 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:42.386039 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:42.386045 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:42.386484 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:42.386666 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:42.388024 (+ 1358us) server_negotiation.cc:299] Negotiation successful
0926 02:07:42.388415 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19616,"thread_start_us":19460,"threads_started":1}
I20260926 02:07:42.392779 26493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.345498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:43111 (local address 127.18.159.131:54089)
0926 02:07:42.359655 (+ 14157us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:42.359749 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:42.363416 (+ 3667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:42.365864 (+ 2448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:42.365881 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:42.366858 (+ 977us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:42.367897 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:42.367928 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.370246 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.370254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:42.372231 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:42.372246 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.373655 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.373666 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:42.374000 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.374880 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:42.374925 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:42.378398 (+ 3473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:42.383969 (+ 5571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:42.383984 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:42.384012 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:42.384933 (+ 921us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:42.385522 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:42.385530 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:42.385536 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:42.385879 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:42.386790 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:42.386803 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:42.387403 (+ 600us) client_negotiation.cc:769] Sending connection context
0926 02:07:42.389824 (+ 2421us) client_negotiation.cc:240] Negotiation successful
0926 02:07:42.390650 (+ 826us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13684,"spinlock_wait_cycles":26880,"thread_start_us":13337,"threads_started":1}
Sep 26 02:07:42 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388462, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM
I20260926 02:07:42.396754 26488 heartbeater.cc:347] Connected to a master server at 127.18.159.189:43111
I20260926 02:07:42.397220 26488 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:42.398332 26488 heartbeater.cc:511] Master 127.18.159.189:43111 requested a full tablet report, sending...
I20260926 02:07:42.400791 25830 ts_manager.cc:194] Registered new tserver with Master: f9a809af0ebb4c5b8b485259aeb2fb1c (127.18.159.131:41509)
Sep 26 02:07:42 dist-test-slave-03rn krb5kdc[25179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388462, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.188@KRBTEST.COM
I20260926 02:07:42.411283 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.344648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:54331 (local address 127.18.159.190:35357)
0926 02:07:42.358381 (+ 13733us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:42.358391 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:42.369264 (+ 10873us) server_negotiation.cc:407] Connection header received
0926 02:07:42.369447 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:42.369455 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:42.369559 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:42.369686 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:42.373615 (+ 3929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.374820 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.378673 (+ 3853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.379392 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.379545 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.401818 (+ 22273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:42.401829 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:42.401837 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:42.401899 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:42.405906 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:42.406997 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:42.407005 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:42.407011 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:42.407188 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:42.409794 (+ 2606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:42.409799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:42.409803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:42.410104 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:42.410274 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:42.410689 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 02:07:42.410923 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13536,"thread_start_us":13555,"threads_started":1}
I20260926 02:07:42.411324 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.345498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.131:54331)
0926 02:07:42.369141 (+ 23643us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:42.369169 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:42.369322 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:42.372259 (+ 2937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:42.372267 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:42.373022 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:42.373418 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:42.373435 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.376372 (+ 2937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.376380 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:42.378474 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:42.378495 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.379600 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.379607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:42.379821 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.380459 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:42.380483 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:42.401450 (+ 20967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:42.406059 (+ 4609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:42.406068 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:42.406075 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:42.406808 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:42.408738 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:42.408747 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:42.408754 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:42.408997 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:42.410241 (+ 1244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:42.410246 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:42.410428 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 02:07:42.410576 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 02:07:42.410886 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23429,"mutex_wait_us":548,"thread_start_us":23963,"threads_started":1}
I20260926 02:07:42.414732 26487 heartbeater.cc:347] Connected to a master server at 127.18.159.190:35357
I20260926 02:07:42.414906 26487 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:42.415450 26487 heartbeater.cc:511] Master 127.18.159.190:35357 requested a full tablet report, sending...
I20260926 02:07:42.416669 25223 ts_manager.cc:194] Re-registered known tserver with Master: f9a809af0ebb4c5b8b485259aeb2fb1c (127.18.159.131:41509)
I20260926 02:07:42.417887 25223 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:54331
I20260926 02:07:42.424134 26492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.344589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:55925 (local address 127.18.159.188:40445)
0926 02:07:42.358824 (+ 14235us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:42.358833 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:42.366386 (+ 7553us) server_negotiation.cc:407] Connection header received
0926 02:07:42.366695 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:42.366704 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:42.366817 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:42.366925 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:42.369520 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.371875 (+ 2355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.376338 (+ 4463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.377052 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.377210 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.414316 (+ 37106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:42.414327 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:42.414335 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:42.414397 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:42.418475 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:42.421512 (+ 3037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:42.421523 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:42.421529 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:42.421756 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:42.422359 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:42.422366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:42.422369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:42.422732 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:42.422874 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:42.423486 (+ 612us) server_negotiation.cc:299] Negotiation successful
0926 02:07:42.423725 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14060,"thread_start_us":15581,"threads_started":1}
I20260926 02:07:42.424190 26495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.345865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:40445 (local address 127.18.159.131:55925)
0926 02:07:42.366230 (+ 20365us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:42.366261 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:42.366518 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:42.367211 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:42.367219 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:42.368079 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:42.368485 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:42.368501 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.373702 (+ 5201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.373710 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:42.376116 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:42.376140 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.377277 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.377285 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:42.377543 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.378273 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:42.378328 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:42.412329 (+ 34001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:42.418699 (+ 6370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:42.418709 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:42.418716 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:42.419567 (+ 851us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:42.421972 (+ 2405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:42.421980 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:42.421987 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:42.422192 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:42.423023 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:42.423027 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:42.423198 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 02:07:42.423355 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 02:07:42.423702 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20171,"mutex_wait_us":4,"spinlock_wait_cycles":23168,"thread_start_us":23995,"threads_started":1}
I20260926 02:07:42.425149 26486 heartbeater.cc:347] Connected to a master server at 127.18.159.188:40445
I20260926 02:07:42.425305 26486 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:42.425763 26486 heartbeater.cc:511] Master 127.18.159.188:40445 requested a full tablet report, sending...
I20260926 02:07:42.427062 26023 ts_manager.cc:194] Registered new tserver with Master: f9a809af0ebb4c5b8b485259aeb2fb1c (127.18.159.131:41509)
I20260926 02:07:42.428504 26490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.381103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37834 (local address 127.18.159.189:43111)
0926 02:07:42.389695 (+ 8592us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:42.389708 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:42.402713 (+ 13005us) server_negotiation.cc:407] Connection header received
0926 02:07:42.403232 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:42.403238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:42.403353 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:42.403428 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:42.406006 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.406727 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.408922 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.409444 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.409520 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.417940 (+ 8420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:42.417948 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:42.417953 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:42.418001 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:42.421887 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:42.424220 (+ 2333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:42.424228 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:42.424232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:42.424403 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:42.425366 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:42.425375 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:42.425379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:42.425777 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:42.426074 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:42.427815 (+ 1741us) server_negotiation.cc:299] Negotiation successful
0926 02:07:42.428096 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8445,"thread_start_us":41900,"threads_started":1}
I20260926 02:07:42.433830 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.382022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55616 (local address 127.18.159.190:35357)
0926 02:07:42.411736 (+ 29714us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:42.411742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:42.411762 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:07:42.411824 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:42.411832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:42.411912 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:42.412017 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:42.414564 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.416080 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.422066 (+ 5986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.422653 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.422798 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.426927 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:42.426934 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:42.426940 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:42.426980 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:42.430601 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:42.431549 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:42.431557 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:42.431562 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:42.431733 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:42.432268 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:42.432276 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:42.432282 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:42.432654 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:42.432925 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:42.433279 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 02:07:42.433506 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29605,"thread_start_us":39259,"threads_started":1}
I20260926 02:07:42.441354 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.382872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58634 (local address 127.18.159.188:40445)
0926 02:07:42.398602 (+ 15730us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:42.398608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:42.422944 (+ 24336us) server_negotiation.cc:407] Connection header received
0926 02:07:42.424790 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:42.424794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:42.424871 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:42.424957 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:42.427764 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.428940 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.431172 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.431751 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.431887 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.435006 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:42.435013 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:42.435018 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:42.435059 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:42.438338 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:42.439222 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:42.439228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:42.439232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:42.439382 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:42.439779 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:42.439785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:42.439788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:42.440092 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:42.440360 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:42.440736 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 02:07:42.440982 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15570,"thread_start_us":15737,"threads_started":1}
I20260926 02:07:42.439035 25223 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:07:42.447744 25223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:07:42.485460 26518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.469691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:41509 (local address 127.0.0.1:47840)
0926 02:07:42.471738 (+ 2047us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:42.471764 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:42.471951 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:42.473519 (+ 1568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:42.473527 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:42.473568 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:42.474165 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:42.474190 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.477264 (+ 3074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.477273 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:42.480753 (+ 3480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:42.480788 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.484147 (+ 3359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.484198 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:42.484507 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.484582 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 02:07:42.484708 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 02:07:42.484933 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1859,"threads_started":1}
I20260926 02:07:42.487069 26517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.470074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47840 (local address 127.18.159.131:41509)
0926 02:07:42.470929 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:42.470939 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:42.472276 (+ 1337us) server_negotiation.cc:407] Connection header received
0926 02:07:42.472440 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:42.472454 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:42.472856 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:42.473112 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:42.474506 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.476980 (+ 2474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.481076 (+ 4096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.483405 (+ 2329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.483678 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.483940 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:42.486270 (+ 2330us) server_negotiation.cc:299] Negotiation successful
0926 02:07:42.486584 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":435,"threads_started":1}
I20260926 02:07:42.497334 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.471711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57106 (local address 127.18.159.130:42645)
0926 02:07:42.488692 (+ 16981us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:42.488705 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:42.488751 (+ 46us) server_negotiation.cc:407] Connection header received
0926 02:07:42.488888 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:42.488901 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:42.489280 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:42.489587 (+ 307us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:42.490590 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.492316 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.494616 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.496368 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.496566 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.496773 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:42.496892 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 02:07:42.497061 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16813,"thread_start_us":16707,"threads_started":1}
I20260926 02:07:42.497386 26518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.471440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:42645 (local address 127.0.0.1:57106)
0926 02:07:42.485820 (+ 14380us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:42.485851 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:42.486005 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:42.490198 (+ 4193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:42.490204 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:42.490240 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:42.490402 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:42.490410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.492475 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.492481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:42.494395 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:42.494416 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.496535 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.496543 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:42.496715 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.496754 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 02:07:42.496846 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 02:07:42.497015 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14209,"mutex_wait_us":1,"thread_start_us":8809,"threads_started":1}
I20260926 02:07:42.496765 26421 tablet_service.cc:1467] Processing CreateTablet for tablet 41982a7c8dbc49f2a2925c20c5facf9d (DEFAULT_TABLE table=test-table [id=b90741f67cbe4ba483303b406818d70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:42.498445 26421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41982a7c8dbc49f2a2925c20c5facf9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:42.499045 26520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.470986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:34533 (local address 127.0.0.1:49812)
0926 02:07:42.481337 (+ 10351us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:42.481363 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:42.481532 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:42.491269 (+ 9737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:42.491274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:42.491304 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:42.492220 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:42.492241 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.493696 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.493702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:42.495919 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:42.495937 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.498180 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.498188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:42.498381 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.498423 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 02:07:42.498530 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 02:07:42.498688 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10161,"thread_start_us":654,"threads_started":1}
I20260926 02:07:42.499439 26519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.471167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49812 (local address 127.18.159.129:34533)
0926 02:07:42.490387 (+ 19220us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:42.490401 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:42.490445 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:07:42.490622 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:42.490635 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:42.490986 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:42.491261 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:42.492433 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.493526 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.496121 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.497979 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.498154 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.498325 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:42.498703 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 02:07:42.498920 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18932,"thread_start_us":18748,"threads_started":1}
I20260926 02:07:42.501443 26280 tablet_service.cc:1467] Processing CreateTablet for tablet 41982a7c8dbc49f2a2925c20c5facf9d (DEFAULT_TABLE table=test-table [id=b90741f67cbe4ba483303b406818d70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:42.502719 26280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41982a7c8dbc49f2a2925c20c5facf9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:42.503296 26140 tablet_service.cc:1467] Processing CreateTablet for tablet 41982a7c8dbc49f2a2925c20c5facf9d (DEFAULT_TABLE table=test-table [id=b90741f67cbe4ba483303b406818d70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:42.505195 26140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41982a7c8dbc49f2a2925c20c5facf9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:42.515756 26522 tablet_bootstrap.cc:492] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284: Bootstrap starting.
I20260926 02:07:42.518375 26523 tablet_bootstrap.cc:492] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c: Bootstrap starting.
I20260926 02:07:42.521137 26522 tablet_bootstrap.cc:654] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:42.522150 26523 tablet_bootstrap.cc:654] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:42.522411 26524 tablet_bootstrap.cc:492] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66: Bootstrap starting.
I20260926 02:07:42.522521 26522 log.cc:824] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:42.523437 26523 log.cc:824] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:42.526916 26524 tablet_bootstrap.cc:654] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:42.527390 26522 tablet_bootstrap.cc:492] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284: No bootstrap required, opened a new log
I20260926 02:07:42.527491 26523 tablet_bootstrap.cc:492] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c: No bootstrap required, opened a new log
I20260926 02:07:42.527693 26522 ts_tablet_manager.cc:1397] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20260926 02:07:42.527760 26523 ts_tablet_manager.cc:1397] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260926 02:07:42.530172 26524 log.cc:824] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:42.534260 26524 tablet_bootstrap.cc:492] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66: No bootstrap required, opened a new log
I20260926 02:07:42.534615 26524 ts_tablet_manager.cc:1397] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.007s
I20260926 02:07:42.536285 26523 raft_consensus.cc:348] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 } } peers { permanent_uuid: "17e22f81cc54435fa3db75d74f44e284" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 42645 } } peers { permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 } }
I20260926 02:07:42.536684 26523 raft_consensus.cc:374] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:42.536789 26523 raft_consensus.cc:729] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a809af0ebb4c5b8b485259aeb2fb1c, State: Initialized, Role: FOLLOWER
I20260926 02:07:42.537606 26523 consensus_queue.cc:260] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 } } peers { permanent_uuid: "17e22f81cc54435fa3db75d74f44e284" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 42645 } } peers { permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 } }
I20260926 02:07:42.537869 26522 raft_consensus.cc:348] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 } } peers { permanent_uuid: "17e22f81cc54435fa3db75d74f44e284" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 42645 } } peers { permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 } }
I20260926 02:07:42.538337 26522 raft_consensus.cc:374] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:42.538923 26522 raft_consensus.cc:729] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17e22f81cc54435fa3db75d74f44e284, State: Initialized, Role: FOLLOWER
I20260926 02:07:42.539510 26523 ts_tablet_manager.cc:1428] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260926 02:07:42.539817 26487 heartbeater.cc:503] Master 127.18.159.190:35357 was elected leader, sending a full tablet report...
I20260926 02:07:42.539585 26522 consensus_queue.cc:260] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 } } peers { permanent_uuid: "17e22f81cc54435fa3db75d74f44e284" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 42645 } } peers { permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 } }
I20260926 02:07:42.545876 26522 ts_tablet_manager.cc:1428] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
I20260926 02:07:42.546250 26346 heartbeater.cc:503] Master 127.18.159.190:35357 was elected leader, sending a full tablet report...
I20260926 02:07:42.547890 26524 raft_consensus.cc:348] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 } } peers { permanent_uuid: "17e22f81cc54435fa3db75d74f44e284" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 42645 } } peers { permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 } }
I20260926 02:07:42.548758 26524 raft_consensus.cc:374] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:42.548942 26524 raft_consensus.cc:729] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f30c24fe8b7457ba6f65e5ef22fee66, State: Initialized, Role: FOLLOWER
I20260926 02:07:42.549996 26524 consensus_queue.cc:260] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 } } peers { permanent_uuid: "17e22f81cc54435fa3db75d74f44e284" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 42645 } } peers { permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 } }
I20260926 02:07:42.552584 26524 ts_tablet_manager.cc:1428] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66: Time spent starting tablet: real 0.018s user 0.009s sys 0.008s
W20260926 02:07:42.570444 26208 tablet.cc:2369] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:07:42.615322 26489 tablet.cc:2369] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:07:42.695218 26348 tablet.cc:2369] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:07:42.704496 26529 raft_consensus.cc:482] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:07:42.704856 26529 raft_consensus.cc:504] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 } } peers { permanent_uuid: "17e22f81cc54435fa3db75d74f44e284" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 42645 } } peers { permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 } }
I20260926 02:07:42.706636 26529 leader_election.cc:288] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9a809af0ebb4c5b8b485259aeb2fb1c (127.18.159.131:41509), 3f30c24fe8b7457ba6f65e5ef22fee66 (127.18.159.129:34533)
I20260926 02:07:42.714481 26519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.707250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:60427 (local address 127.18.159.129:34533)
0926 02:07:42.707447 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:42.707453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:42.708687 (+ 1234us) server_negotiation.cc:407] Connection header received
0926 02:07:42.708804 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:42.708809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:42.708883 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:42.708960 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:42.709693 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.710497 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.712591 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.713389 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.713521 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.713646 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:42.714070 (+ 424us) server_negotiation.cc:299] Negotiation successful
0926 02:07:42.714214 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 02:07:42.714484 26517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.707101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:56405 (local address 127.18.159.131:41509)
0926 02:07:42.707326 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:42.707334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:42.708626 (+ 1292us) server_negotiation.cc:407] Connection header received
0926 02:07:42.708873 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:42.708877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:42.708980 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:42.709101 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:42.709753 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.710508 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.712614 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.713429 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.713642 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.713715 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:42.714076 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 02:07:42.714225 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:07:42.714661 26531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.706933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:34533 (local address 127.18.159.130:60427)
0926 02:07:42.708463 (+ 1530us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:42.708485 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:42.708639 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:42.709023 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:42.709029 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:42.709062 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:42.709489 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:42.709503 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.710667 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.710672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:42.712387 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:42.712405 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.713656 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.713662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:42.713874 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.713923 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 02:07:42.714041 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 02:07:42.714201 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1352,"mutex_wait_us":44,"thread_start_us":608,"threads_started":1}
I20260926 02:07:42.715101 26532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.707146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:41509 (local address 127.18.159.130:56405)
0926 02:07:42.708481 (+ 1335us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:42.708497 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:42.708676 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:42.709153 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:42.709156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:42.709181 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:42.709540 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:42.709549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.710660 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.710667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:42.712393 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:42.712408 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.713714 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.713719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:42.713874 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.713924 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 02:07:42.714037 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 02:07:42.714201 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1225,"mutex_wait_us":19,"thread_start_us":496,"threads_started":1}
I20260926 02:07:42.715831 26160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41982a7c8dbc49f2a2925c20c5facf9d" candidate_uuid: "17e22f81cc54435fa3db75d74f44e284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" is_pre_election: true
I20260926 02:07:42.716089 26441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41982a7c8dbc49f2a2925c20c5facf9d" candidate_uuid: "17e22f81cc54435fa3db75d74f44e284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" is_pre_election: true
I20260926 02:07:42.716480 26160 raft_consensus.cc:2454] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17e22f81cc54435fa3db75d74f44e284 in term 0.
I20260926 02:07:42.716675 26441 raft_consensus.cc:2454] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17e22f81cc54435fa3db75d74f44e284 in term 0.
I20260926 02:07:42.717315 26236 leader_election.cc:302] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17e22f81cc54435fa3db75d74f44e284, 3f30c24fe8b7457ba6f65e5ef22fee66, f9a809af0ebb4c5b8b485259aeb2fb1c; no voters:
I20260926 02:07:42.717839 26529 raft_consensus.cc:2781] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:07:42.717996 26529 raft_consensus.cc:482] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:07:42.718101 26529 raft_consensus.cc:3037] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:42.720394 26529 raft_consensus.cc:504] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 } } peers { permanent_uuid: "17e22f81cc54435fa3db75d74f44e284" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 42645 } } peers { permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 } }
I20260926 02:07:42.720988 26529 leader_election.cc:288] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 [CANDIDATE]: Term 1 election: Requested vote from peers f9a809af0ebb4c5b8b485259aeb2fb1c (127.18.159.131:41509), 3f30c24fe8b7457ba6f65e5ef22fee66 (127.18.159.129:34533)
I20260926 02:07:42.721565 26441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41982a7c8dbc49f2a2925c20c5facf9d" candidate_uuid: "17e22f81cc54435fa3db75d74f44e284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c"
I20260926 02:07:42.721765 26441 raft_consensus.cc:3037] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:42.722011 26160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41982a7c8dbc49f2a2925c20c5facf9d" candidate_uuid: "17e22f81cc54435fa3db75d74f44e284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66"
I20260926 02:07:42.722257 26160 raft_consensus.cc:3037] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:42.725001 26441 raft_consensus.cc:2454] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17e22f81cc54435fa3db75d74f44e284 in term 1.
I20260926 02:07:42.725524 26160 raft_consensus.cc:2454] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17e22f81cc54435fa3db75d74f44e284 in term 1.
I20260926 02:07:42.725721 26235 leader_election.cc:302] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17e22f81cc54435fa3db75d74f44e284, f9a809af0ebb4c5b8b485259aeb2fb1c; no voters:
I20260926 02:07:42.726140 26529 raft_consensus.cc:2781] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:42.726598 26529 raft_consensus.cc:686] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 [term 1 LEADER]: Becoming Leader. State: Replica: 17e22f81cc54435fa3db75d74f44e284, State: Running, Role: LEADER
I20260926 02:07:42.727270 26529 consensus_queue.cc:237] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 } } peers { permanent_uuid: "17e22f81cc54435fa3db75d74f44e284" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 42645 } } peers { permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 } }
I20260926 02:07:42.733996 25223 catalog_manager.cc:5495] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17e22f81cc54435fa3db75d74f44e284 (127.18.159.130). New cstate: current_term: 1 leader_uuid: "17e22f81cc54435fa3db75d74f44e284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17e22f81cc54435fa3db75d74f44e284" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 42645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 } health_report { overall_health: UNKNOWN } } }
I20260926 02:07:42.794584 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.788339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57116 (local address 127.18.159.130:42645)
0926 02:07:42.788547 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:42.788554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:42.788599 (+ 45us) server_negotiation.cc:407] Connection header received
0926 02:07:42.788866 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:42.788870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:42.788986 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:42.789153 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:07:42.789516 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.790263 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.791640 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.792624 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.792778 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.793356 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:07:42.793925 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:07:42.794114 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:42.794202 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 02:07:42.794293 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:07:42.803423 26160 raft_consensus.cc:1261] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66 [term 1 FOLLOWER]: Refusing update from remote peer 17e22f81cc54435fa3db75d74f44e284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:42.803426 26441 raft_consensus.cc:1261] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c [term 1 FOLLOWER]: Refusing update from remote peer 17e22f81cc54435fa3db75d74f44e284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:42.804601 26533 consensus_queue.cc:1035] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:42.804965 26529 consensus_queue.cc:1035] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:42.819478 26537 mvcc.cc:205] Tried to move back new op lower bound from 7333431143632605184 to 7333431143335444480. Current Snapshot: MvccSnapshot[applied={T|T < 7333431143632605184}]
I20260926 02:07:42.843807 25223 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:42.844111 25223 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:42.851055 25223 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16268d1d4f86406b9faa8950ac12a76d: Sending DeleteTablet for 3 replicas of tablet 41982a7c8dbc49f2a2925c20c5facf9d
I20260926 02:07:42.852778 26421 tablet_service.cc:1514] Processing DeleteTablet for tablet 41982a7c8dbc49f2a2925c20c5facf9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:42 UTC) from {username='oryx', principal='oryx/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:47840
I20260926 02:07:42.853579 26280 tablet_service.cc:1514] Processing DeleteTablet for tablet 41982a7c8dbc49f2a2925c20c5facf9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:42 UTC) from {username='oryx', principal='oryx/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:57106
I20260926 02:07:42.853626 26140 tablet_service.cc:1514] Processing DeleteTablet for tablet 41982a7c8dbc49f2a2925c20c5facf9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:42 UTC) from {username='oryx', principal='oryx/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:49812
I20260926 02:07:42.864511 26548 tablet_replica.cc:330] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c: stopping tablet replica
I20260926 02:07:42.865020 26548 raft_consensus.cc:2229] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:42.865677 26548 raft_consensus.cc:2258] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:42.866653 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 25192
I20260926 02:07:42.868101 25254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:42.868770 26546 tablet_replica.cc:330] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66: stopping tablet replica
I20260926 02:07:42.869371 26546 raft_consensus.cc:2229] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:42.870044 26547 tablet_replica.cc:330] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284: stopping tablet replica
I20260926 02:07:42.870656 26546 raft_consensus.cc:2258] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:42.871213 26547 raft_consensus.cc:2229] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:07:42.872336 26547 raft_consensus.cc:2258] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:42.874400 26546 ts_tablet_manager.cc:1905] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:42.874801 26547 ts_tablet_manager.cc:1905] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:42.882500 26548 ts_tablet_manager.cc:1905] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:42.882994 26546 ts_tablet_manager.cc:1918] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:07:42.883179 26546 log.cc:1196] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/41982a7c8dbc49f2a2925c20c5facf9d
I20260926 02:07:42.883419 26547 ts_tablet_manager.cc:1918] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:07:42.883662 26547 log.cc:1196] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/41982a7c8dbc49f2a2925c20c5facf9d
I20260926 02:07:42.883764 26546 ts_tablet_manager.cc:1939] T 41982a7c8dbc49f2a2925c20c5facf9d P 3f30c24fe8b7457ba6f65e5ef22fee66: Deleting consensus metadata
I20260926 02:07:42.884435 26547 ts_tablet_manager.cc:1939] T 41982a7c8dbc49f2a2925c20c5facf9d P 17e22f81cc54435fa3db75d74f44e284: Deleting consensus metadata
I20260926 02:07:42.894913 26548 ts_tablet_manager.cc:1918] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:07:42.895133 26548 log.cc:1196] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/41982a7c8dbc49f2a2925c20c5facf9d
I20260926 02:07:42.895774 26548 ts_tablet_manager.cc:1939] T 41982a7c8dbc49f2a2925c20c5facf9d P f9a809af0ebb4c5b8b485259aeb2fb1c: Deleting consensus metadata
I20260926 02:07:42.900352 26490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.856422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37848 (local address 127.18.159.189:43111)
0926 02:07:42.856591 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:42.856598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:42.863467 (+ 6869us) server_negotiation.cc:407] Connection header received
0926 02:07:42.866950 (+ 3483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:42.866956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:42.867063 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:42.867178 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:42.885753 (+ 18575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.886515 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.888291 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.888940 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.889066 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.893450 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:42.893459 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:42.893464 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:42.893515 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:42.897212 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:42.898180 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:42.898186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:42.898190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:42.898334 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:42.898768 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:42.898774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:42.898779 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:42.899078 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:42.899348 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:42.899710 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 02:07:42.899934 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 02:07:42.913874 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:42.863535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58648 (local address 127.18.159.188:40445)
0926 02:07:42.863807 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:42.863817 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:42.900525 (+ 36708us) server_negotiation.cc:407] Connection header received
0926 02:07:42.900797 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:42.900804 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:42.900896 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:42.900965 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:42.902941 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.903657 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.904986 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:42.905435 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:42.905575 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:42.908263 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:42.908268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:42.908271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:42.908307 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:42.910942 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:42.911640 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:42.911653 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:42.911655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:42.911788 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:42.912285 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:42.912293 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:42.912298 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:42.912595 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:42.912769 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:42.913288 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 02:07:42.913533 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:07:43.017429 25211 catalog_manager.cc:4841] TS 3f30c24fe8b7457ba6f65e5ef22fee66 (127.18.159.129:34533): tablet 41982a7c8dbc49f2a2925c20c5facf9d (table test-table [id=b90741f67cbe4ba483303b406818d70b]) successfully deleted
I20260926 02:07:43.017881 25210 catalog_manager.cc:4841] TS f9a809af0ebb4c5b8b485259aeb2fb1c (127.18.159.131:41509): tablet 41982a7c8dbc49f2a2925c20c5facf9d (table test-table [id=b90741f67cbe4ba483303b406818d70b]) successfully deleted
I20260926 02:07:43.017975 25209 catalog_manager.cc:4841] TS 17e22f81cc54435fa3db75d74f44e284 (127.18.159.130:42645): tablet 41982a7c8dbc49f2a2925c20c5facf9d (table test-table [id=b90741f67cbe4ba483303b406818d70b]) successfully deleted
I20260926 02:07:43.018481 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 25192
I20260926 02:07:43.085260 26490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:43.048341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37852 (local address 127.18.159.189:43111)
0926 02:07:43.048580 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:43.048590 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:43.061551 (+ 12961us) server_negotiation.cc:407] Connection header received
0926 02:07:43.061974 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:43.061984 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:43.062098 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:43.062199 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:43.068372 (+ 6173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:43.069472 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:43.073083 (+ 3611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:43.073802 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:43.073902 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:43.078189 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:43.078196 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:43.078201 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:43.078250 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:43.081977 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:43.083011 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:43.083018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:43.083021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:43.083184 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:43.083662 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:43.083667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:43.083670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:43.083984 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:43.084279 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:43.084696 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 02:07:43.084916 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:07:43.086236 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:43.051781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58662 (local address 127.18.159.188:40445)
0926 02:07:43.051975 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:43.051982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:43.062462 (+ 10480us) server_negotiation.cc:407] Connection header received
0926 02:07:43.062730 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:43.062736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:43.062818 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:43.062888 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:43.066492 (+ 3604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:43.067582 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:43.072980 (+ 5398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:43.073672 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:43.073792 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:43.078201 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:43.078209 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:43.078213 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:43.078281 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:43.082210 (+ 3929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:43.083356 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:43.083361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:43.083365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:43.083508 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:43.084620 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:43.084625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:43.084629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:43.084963 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:43.085174 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:43.085702 (+ 528us) server_negotiation.cc:299] Negotiation successful
0926 02:07:43.085919 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 02:07:43.832569 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:43.831826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.130:34999)
0926 02:07:43.832281 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:43.832400 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":202,"threads_started":1}
W20260926 02:07:43.833016 26346 heartbeater.cc:644] Failed to heartbeat to 127.18.159.190:35357 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
I20260926 02:07:43.836946 26559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:43.836037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.129:44007)
0926 02:07:43.836615 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:43.836733 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":313,"threads_started":1}
W20260926 02:07:43.837409 26206 heartbeater.cc:644] Failed to heartbeat to 127.18.159.190:35357 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
I20260926 02:07:43.839464 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:43.838602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.131:53215)
0926 02:07:43.839083 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:43.839224 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":249,"threads_started":1}
W20260926 02:07:43.839862 26487 heartbeater.cc:644] Failed to heartbeat to 127.18.159.190:35357 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
I20260926 02:07:44.352470 26561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 16268d1d4f86406b9faa8950ac12a76d)
I20260926 02:07:44.352325 26562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 16268d1d4f86406b9faa8950ac12a76d)
I20260926 02:07:44.352955 26562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: false } }
I20260926 02:07:44.352960 26561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: false } }
I20260926 02:07:44.354352 26561 leader_election.cc:288] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16268d1d4f86406b9faa8950ac12a76d (127.18.159.190:35357), 04c3c25b95fe45f48ba2a2ab12765cf9 (127.18.159.188:40445)
I20260926 02:07:44.360949 26562 leader_election.cc:288] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16268d1d4f86406b9faa8950ac12a76d (127.18.159.190:35357), 4d7e2176af0a475eab3956ac474f6320 (127.18.159.189:43111)
I20260926 02:07:44.361845 26564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.354503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.0.0.1:56058)
0926 02:07:44.361070 (+ 6567us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.361282 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5965,"thread_start_us":5803,"threads_started":1}
I20260926 02:07:44.362488 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.354503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.0.0.1:56042)
0926 02:07:44.361758 (+ 7255us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.362117 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7133,"thread_start_us":7091,"threads_started":1}
W20260926 02:07:44.362459 26007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
W20260926 02:07:44.363459 25814 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
I20260926 02:07:44.376013 26569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.372746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.0.0.1:56062)
0926 02:07:44.375281 (+ 2535us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.375583 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2355,"thread_start_us":1943,"threads_started":1}
I20260926 02:07:44.376411 26565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.375305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.0.0.1:56076)
0926 02:07:44.375820 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.376047 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339}
W20260926 02:07:44.376648 25814 leader_election.cc:334] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 16268d1d4f86406b9faa8950ac12a76d (127.18.159.190:35357): Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
W20260926 02:07:44.378189 26007 leader_election.cc:334] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 16268d1d4f86406b9faa8950ac12a76d (127.18.159.190:35357): Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
I20260926 02:07:44.380370 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.354589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:40445 (local address 127.0.0.1:58678)
0926 02:07:44.362767 (+ 8178us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.362791 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:44.363024 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:44.370303 (+ 7279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:44.370310 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:44.370345 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:44.370588 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:44.370603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.373348 (+ 2745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.373357 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:44.377008 (+ 3651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:44.377031 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.379362 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.379369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:44.379598 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:44.379642 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 02:07:44.379754 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 02:07:44.379975 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8081,"thread_start_us":15125,"threads_started":1}
I20260926 02:07:44.380558 26566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.355235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58678 (local address 127.18.159.188:40445)
0926 02:07:44.369682 (+ 14447us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:44.369689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:44.369716 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:07:44.369816 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:44.369821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:44.369914 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:44.370040 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:44.370796 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.373108 (+ 2312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.377335 (+ 4227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.379057 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.379857 (+ 800us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:44.379968 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:44.380024 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 02:07:44.380195 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":567,"server-negotiator.queue_time_us":14276,"thread_start_us":14705,"threads_started":1}
I20260926 02:07:44.381276 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d7e2176af0a475eab3956ac474f6320" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" is_pre_election: true
I20260926 02:07:44.381767 26033 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d7e2176af0a475eab3956ac474f6320 in term 1.
I20260926 02:07:44.382684 25817 leader_election.cc:302] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04c3c25b95fe45f48ba2a2ab12765cf9, 4d7e2176af0a475eab3956ac474f6320; no voters: 16268d1d4f86406b9faa8950ac12a76d
I20260926 02:07:44.383180 26561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 02:07:44.383322 26561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 16268d1d4f86406b9faa8950ac12a76d)
I20260926 02:07:44.383441 26561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:07:44.386322 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.372825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.18.159.189:43111)
0926 02:07:44.378574 (+ 5749us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:44.378580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:44.378606 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:07:44.378672 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:44.378677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:44.378788 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:44.379225 (+ 437us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:44.379745 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.381361 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.384049 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.385158 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.385316 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:44.385376 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:44.385914 (+ 538us) server_negotiation.cc:299] Negotiation successful
0926 02:07:44.386040 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5511,"thread_start_us":5201,"threads_started":1}
I20260926 02:07:44.386330 26564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.370316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:43111 (local address 127.0.0.1:37862)
0926 02:07:44.370589 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.370611 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:44.370729 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:44.378987 (+ 8258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:44.378992 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:44.379087 (+ 95us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:44.379528 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:44.379541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.381548 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.381553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:44.383839 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:44.383860 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.385395 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.385404 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:44.385680 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:44.385729 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 02:07:44.385851 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 02:07:44.386021 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":42}
I20260926 02:07:44.386722 26561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: false } }
I20260926 02:07:44.387300 25840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4d7e2176af0a475eab3956ac474f6320" is_pre_election: true
I20260926 02:07:44.387616 26561 leader_election.cc:288] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [CANDIDATE]: Term 2 election: Requested vote from peers 16268d1d4f86406b9faa8950ac12a76d (127.18.159.190:35357), 04c3c25b95fe45f48ba2a2ab12765cf9 (127.18.159.188:40445)
I20260926 02:07:44.387733 25840 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 04c3c25b95fe45f48ba2a2ab12765cf9 in current term 2: Already voted for candidate 4d7e2176af0a475eab3956ac474f6320 in this term.
I20260926 02:07:44.388209 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.387658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.0.0.1:56090)
0926 02:07:44.387892 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.387971 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 02:07:44.388263 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d7e2176af0a475eab3956ac474f6320" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9"
I20260926 02:07:44.388535 26033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:07:44.388474 26008 leader_election.cc:302] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 04c3c25b95fe45f48ba2a2ab12765cf9; no voters: 16268d1d4f86406b9faa8950ac12a76d, 4d7e2176af0a475eab3956ac474f6320
I20260926 02:07:44.389283 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.388947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.0.0.1:56102)
0926 02:07:44.389086 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.389181 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 02:07:44.389528 25814 leader_election.cc:334] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 16268d1d4f86406b9faa8950ac12a76d (127.18.159.190:35357): Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
I20260926 02:07:44.390964 26033 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d7e2176af0a475eab3956ac474f6320 in term 2.
I20260926 02:07:44.391173 26562 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 02:07:44.391522 25817 leader_election.cc:302] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04c3c25b95fe45f48ba2a2ab12765cf9, 4d7e2176af0a475eab3956ac474f6320; no voters: 16268d1d4f86406b9faa8950ac12a76d
I20260926 02:07:44.391916 26561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 02:07:44.392668 26561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [term 2 LEADER]: Becoming Leader. State: Replica: 4d7e2176af0a475eab3956ac474f6320, State: Running, Role: LEADER
I20260926 02:07:44.393100 26561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: false } }
I20260926 02:07:44.394968 26571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d7e2176af0a475eab3956ac474f6320. Latest consensus state: current_term: 2 leader_uuid: "4d7e2176af0a475eab3956ac474f6320" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: false } } }
I20260926 02:07:44.395431 26571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:44.396426 26573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:07:44.400300 26573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:07:44.401589 26573 catalog_manager.cc:1259] Loaded cluster ID: 1fd63064f7554b2a889942cd3cb66892
I20260926 02:07:44.401782 26573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:07:44.403501 26573 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:07:44.404657 26573 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: Loaded TSK: 0
I20260926 02:07:44.405627 26573 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:07:44.430392 25830 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:07:44.438448 25830 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:07:44.443318 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.442810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.0.0.1:56114)
0926 02:07:44.443017 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.443123 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 02:07:44.443336 26033 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [term 2 FOLLOWER]: Refusing update from remote peer 4d7e2176af0a475eab3956ac474f6320: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 02:07:44.444193 26571 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 02:07:44.444535 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.444101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.0.0.1:56122)
0926 02:07:44.444308 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.444382 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260926 02:07:44.444746 25814 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 -> Peer 16268d1d4f86406b9faa8950ac12a76d (127.18.159.190:35357): Couldn't send request to peer 16268d1d4f86406b9faa8950ac12a76d. Status: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 02:07:44.447567 26562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d7e2176af0a475eab3956ac474f6320. Latest consensus state: current_term: 2 leader_uuid: "4d7e2176af0a475eab3956ac474f6320" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: false } } }
I20260926 02:07:44.447913 26562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:07:44.449777 26561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4d7e2176af0a475eab3956ac474f6320" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: false } } }
I20260926 02:07:44.450157 26561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320 [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:44.450843 26562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4d7e2176af0a475eab3956ac474f6320" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "16268d1d4f86406b9faa8950ac12a76d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35357 } } peers { permanent_uuid: "4d7e2176af0a475eab3956ac474f6320" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 43111 } attrs { promote: false } } peers { permanent_uuid: "04c3c25b95fe45f48ba2a2ab12765cf9" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 40445 } attrs { promote: false } } }
I20260926 02:07:44.451102 26562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c3c25b95fe45f48ba2a2ab12765cf9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:07:44.470741 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.459737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:41509 (local address 127.0.0.1:47848)
0926 02:07:44.459989 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.460012 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:44.460137 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:44.461058 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:44.461065 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:44.461102 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:44.461378 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:44.461395 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.464286 (+ 2891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.464297 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:44.467122 (+ 2825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:44.467149 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.469553 (+ 2404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.469564 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:44.469885 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:44.469947 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 02:07:44.470037 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 02:07:44.470252 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 02:07:44.472939 26579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.459797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47848 (local address 127.18.159.131:41509)
0926 02:07:44.460288 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:44.460295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:44.460346 (+ 51us) server_negotiation.cc:407] Connection header received
0926 02:07:44.460432 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:44.460437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:44.460521 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:44.462227 (+ 1706us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:44.462490 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.463964 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.467477 (+ 3513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.469284 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.472030 (+ 2746us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:44.472177 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:44.472253 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 02:07:44.472479 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":237,"threads_started":1}
I20260926 02:07:44.475466 26421 tablet_service.cc:1467] Processing CreateTablet for tablet 46e25c919f9d41e28d7a3523f0477495 (DEFAULT_TABLE table=test-table [id=3810cf79ca7049438d689039da0b07de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:44.476261 26421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e25c919f9d41e28d7a3523f0477495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:44.483554 26583 tablet_bootstrap.cc:492] T 46e25c919f9d41e28d7a3523f0477495 P f9a809af0ebb4c5b8b485259aeb2fb1c: Bootstrap starting.
I20260926 02:07:44.485415 26565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.460514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:34533 (local address 127.0.0.1:49816)
0926 02:07:44.460664 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.460681 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:44.460800 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:44.478634 (+ 17834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:44.478639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:44.478671 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:44.479009 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:44.479021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.480135 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.480142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:44.482286 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:44.482302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.484149 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.484182 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:44.484822 (+ 640us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:44.484859 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 02:07:44.484978 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 02:07:44.485152 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 02:07:44.485649 26583 tablet_bootstrap.cc:654] T 46e25c919f9d41e28d7a3523f0477495 P f9a809af0ebb4c5b8b485259aeb2fb1c: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:44.486136 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.460847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49816 (local address 127.18.159.129:34533)
0926 02:07:44.478213 (+ 17366us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:44.478219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:44.478254 (+ 35us) server_negotiation.cc:407] Connection header received
0926 02:07:44.478333 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:44.478339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:44.478423 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:44.478573 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:44.479225 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.479973 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.482495 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.483696 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.483874 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:44.483958 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:44.485760 (+ 1802us) server_negotiation.cc:299] Negotiation successful
0926 02:07:44.485915 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17220,"thread_start_us":17106,"threads_started":1}
I20260926 02:07:44.486932 26140 tablet_service.cc:1467] Processing CreateTablet for tablet 46e25c919f9d41e28d7a3523f0477495 (DEFAULT_TABLE table=test-table [id=3810cf79ca7049438d689039da0b07de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:44.487432 26140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e25c919f9d41e28d7a3523f0477495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:44.488062 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.460543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:42645 (local address 127.0.0.1:57128)
0926 02:07:44.471094 (+ 10551us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.471122 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:44.471296 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:44.481404 (+ 10108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:44.481410 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:44.481440 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:44.481651 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:44.481662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.482979 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.482988 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:44.485090 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:44.485108 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.487138 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.487145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:44.487387 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:44.487436 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 02:07:44.487551 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 02:07:44.487696 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10400,"mutex_wait_us":23,"thread_start_us":19846,"threads_started":1}
I20260926 02:07:44.488293 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.460657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.18.159.130:42645)
0926 02:07:44.481045 (+ 20388us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:44.481052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:44.481073 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:07:44.481148 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:44.481155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:44.481239 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:44.481791 (+ 552us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:44.481954 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.482841 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.485464 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.486887 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.487142 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:44.487286 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:44.487609 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 02:07:44.487820 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20256,"thread_start_us":20126,"threads_started":1}
I20260926 02:07:44.489251 26280 tablet_service.cc:1467] Processing CreateTablet for tablet 46e25c919f9d41e28d7a3523f0477495 (DEFAULT_TABLE table=test-table [id=3810cf79ca7049438d689039da0b07de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:44.489830 26280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e25c919f9d41e28d7a3523f0477495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:44.493491 26585 tablet_bootstrap.cc:492] T 46e25c919f9d41e28d7a3523f0477495 P 3f30c24fe8b7457ba6f65e5ef22fee66: Bootstrap starting.
I20260926 02:07:44.495594 26585 tablet_bootstrap.cc:654] T 46e25c919f9d41e28d7a3523f0477495 P 3f30c24fe8b7457ba6f65e5ef22fee66: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:44.496229 26583 tablet_bootstrap.cc:492] T 46e25c919f9d41e28d7a3523f0477495 P f9a809af0ebb4c5b8b485259aeb2fb1c: No bootstrap required, opened a new log
I20260926 02:07:44.496397 26583 ts_tablet_manager.cc:1397] T 46e25c919f9d41e28d7a3523f0477495 P f9a809af0ebb4c5b8b485259aeb2fb1c: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260926 02:07:44.497216 26583 raft_consensus.cc:348] T 46e25c919f9d41e28d7a3523f0477495 P f9a809af0ebb4c5b8b485259aeb2fb1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 } } peers { permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 } } peers { permanent_uuid: "17e22f81cc54435fa3db75d74f44e284" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 42645 } }
I20260926 02:07:44.497467 26583 raft_consensus.cc:374] T 46e25c919f9d41e28d7a3523f0477495 P f9a809af0ebb4c5b8b485259aeb2fb1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:44.497570 26583 raft_consensus.cc:729] T 46e25c919f9d41e28d7a3523f0477495 P f9a809af0ebb4c5b8b485259aeb2fb1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a809af0ebb4c5b8b485259aeb2fb1c, State: Initialized, Role: FOLLOWER
I20260926 02:07:44.497699 26585 tablet_bootstrap.cc:492] T 46e25c919f9d41e28d7a3523f0477495 P 3f30c24fe8b7457ba6f65e5ef22fee66: No bootstrap required, opened a new log
I20260926 02:07:44.497848 26585 ts_tablet_manager.cc:1397] T 46e25c919f9d41e28d7a3523f0477495 P 3f30c24fe8b7457ba6f65e5ef22fee66: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 02:07:44.497839 26583 consensus_queue.cc:260] T 46e25c919f9d41e28d7a3523f0477495 P f9a809af0ebb4c5b8b485259aeb2fb1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 } } peers { permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 } } peers { permanent_uuid: "17e22f81cc54435fa3db75d74f44e284" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 42645 } }
I20260926 02:07:44.498701 26585 raft_consensus.cc:348] T 46e25c919f9d41e28d7a3523f0477495 P 3f30c24fe8b7457ba6f65e5ef22fee66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 } } peers { permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 } } peers { permanent_uuid: "17e22f81cc54435fa3db75d74f44e284" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 42645 } }
I20260926 02:07:44.498901 26583 ts_tablet_manager.cc:1428] T 46e25c919f9d41e28d7a3523f0477495 P f9a809af0ebb4c5b8b485259aeb2fb1c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:07:44.498980 26585 raft_consensus.cc:374] T 46e25c919f9d41e28d7a3523f0477495 P 3f30c24fe8b7457ba6f65e5ef22fee66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:44.499082 26585 raft_consensus.cc:729] T 46e25c919f9d41e28d7a3523f0477495 P 3f30c24fe8b7457ba6f65e5ef22fee66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f30c24fe8b7457ba6f65e5ef22fee66, State: Initialized, Role: FOLLOWER
I20260926 02:07:44.499320 26585 consensus_queue.cc:260] T 46e25c919f9d41e28d7a3523f0477495 P 3f30c24fe8b7457ba6f65e5ef22fee66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 } } peers { permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 } } peers { permanent_uuid: "17e22f81cc54435fa3db75d74f44e284" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 42645 } }
I20260926 02:07:44.503254 26488 heartbeater.cc:503] Master 127.18.159.189:43111 was elected leader, sending a full tablet report...
I20260926 02:07:44.503602 26590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.499811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.131:49665)
0926 02:07:44.503028 (+ 3217us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.503221 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3051,"threads_started":1}
I20260926 02:07:44.504281 26587 tablet_bootstrap.cc:492] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284: Bootstrap starting.
I20260926 02:07:44.504746 26590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.504378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.131:43645)
0926 02:07:44.504522 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.504606 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 02:07:44.505705 26585 ts_tablet_manager.cc:1428] T 46e25c919f9d41e28d7a3523f0477495 P 3f30c24fe8b7457ba6f65e5ef22fee66: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260926 02:07:44.507738 26587 tablet_bootstrap.cc:654] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:44.508183 26207 heartbeater.cc:503] Master 127.18.159.189:43111 was elected leader, sending a full tablet report...
I20260926 02:07:44.511600 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.506267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.129:47881)
0926 02:07:44.511088 (+ 4821us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.511301 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4668,"thread_start_us":4532,"threads_started":1}
I20260926 02:07:44.511973 26587 tablet_bootstrap.cc:492] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284: No bootstrap required, opened a new log
I20260926 02:07:44.512176 26587 ts_tablet_manager.cc:1397] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 02:07:44.512790 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.512391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.129:41359)
0926 02:07:44.512560 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.512650 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 02:07:44.513078 26587 raft_consensus.cc:348] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 } } peers { permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 } } peers { permanent_uuid: "17e22f81cc54435fa3db75d74f44e284" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 42645 } }
I20260926 02:07:44.513271 26587 raft_consensus.cc:374] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:44.513378 26587 raft_consensus.cc:729] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17e22f81cc54435fa3db75d74f44e284, State: Initialized, Role: FOLLOWER
I20260926 02:07:44.513615 26587 consensus_queue.cc:260] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 } } peers { permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 } } peers { permanent_uuid: "17e22f81cc54435fa3db75d74f44e284" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 42645 } }
I20260926 02:07:44.514367 26587 ts_tablet_manager.cc:1428] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:07:44.515942 26594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.514651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.130:48889)
0926 02:07:44.515539 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.515683 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":618,"threads_started":1}
I20260926 02:07:44.516940 26347 heartbeater.cc:503] Master 127.18.159.189:43111 was elected leader, sending a full tablet report...
I20260926 02:07:44.517273 26594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.516781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.130:38581)
0926 02:07:44.516947 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.517062 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 02:07:44.589669 26593 raft_consensus.cc:482] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:07:44.589906 26593 raft_consensus.cc:504] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 } } peers { permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 } } peers { permanent_uuid: "17e22f81cc54435fa3db75d74f44e284" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 42645 } }
I20260926 02:07:44.590771 26593 leader_election.cc:288] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9a809af0ebb4c5b8b485259aeb2fb1c (127.18.159.131:41509), 3f30c24fe8b7457ba6f65e5ef22fee66 (127.18.159.129:34533)
I20260926 02:07:44.591562 26441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e25c919f9d41e28d7a3523f0477495" candidate_uuid: "17e22f81cc54435fa3db75d74f44e284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" is_pre_election: true
I20260926 02:07:44.591571 26160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e25c919f9d41e28d7a3523f0477495" candidate_uuid: "17e22f81cc54435fa3db75d74f44e284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" is_pre_election: true
I20260926 02:07:44.591869 26441 raft_consensus.cc:2454] T 46e25c919f9d41e28d7a3523f0477495 P f9a809af0ebb4c5b8b485259aeb2fb1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17e22f81cc54435fa3db75d74f44e284 in term 0.
I20260926 02:07:44.591858 26160 raft_consensus.cc:2454] T 46e25c919f9d41e28d7a3523f0477495 P 3f30c24fe8b7457ba6f65e5ef22fee66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17e22f81cc54435fa3db75d74f44e284 in term 0.
I20260926 02:07:44.592487 26235 leader_election.cc:302] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17e22f81cc54435fa3db75d74f44e284, f9a809af0ebb4c5b8b485259aeb2fb1c; no voters:
I20260926 02:07:44.592813 26593 raft_consensus.cc:2781] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:07:44.592924 26593 raft_consensus.cc:482] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:07:44.592993 26593 raft_consensus.cc:3037] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:44.595965 26593 raft_consensus.cc:504] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 } } peers { permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 } } peers { permanent_uuid: "17e22f81cc54435fa3db75d74f44e284" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 42645 } }
I20260926 02:07:44.596846 26593 leader_election.cc:288] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 [CANDIDATE]: Term 1 election: Requested vote from peers f9a809af0ebb4c5b8b485259aeb2fb1c (127.18.159.131:41509), 3f30c24fe8b7457ba6f65e5ef22fee66 (127.18.159.129:34533)
I20260926 02:07:44.597388 26441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e25c919f9d41e28d7a3523f0477495" candidate_uuid: "17e22f81cc54435fa3db75d74f44e284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c"
I20260926 02:07:44.597507 26160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e25c919f9d41e28d7a3523f0477495" candidate_uuid: "17e22f81cc54435fa3db75d74f44e284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66"
I20260926 02:07:44.597631 26441 raft_consensus.cc:3037] T 46e25c919f9d41e28d7a3523f0477495 P f9a809af0ebb4c5b8b485259aeb2fb1c [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:44.597775 26160 raft_consensus.cc:3037] T 46e25c919f9d41e28d7a3523f0477495 P 3f30c24fe8b7457ba6f65e5ef22fee66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:44.599781 26441 raft_consensus.cc:2454] T 46e25c919f9d41e28d7a3523f0477495 P f9a809af0ebb4c5b8b485259aeb2fb1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17e22f81cc54435fa3db75d74f44e284 in term 1.
I20260926 02:07:44.599787 26160 raft_consensus.cc:2454] T 46e25c919f9d41e28d7a3523f0477495 P 3f30c24fe8b7457ba6f65e5ef22fee66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17e22f81cc54435fa3db75d74f44e284 in term 1.
I20260926 02:07:44.600340 26236 leader_election.cc:302] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17e22f81cc54435fa3db75d74f44e284, 3f30c24fe8b7457ba6f65e5ef22fee66; no voters:
I20260926 02:07:44.600714 26593 raft_consensus.cc:2781] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:44.601243 26593 raft_consensus.cc:686] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 [term 1 LEADER]: Becoming Leader. State: Replica: 17e22f81cc54435fa3db75d74f44e284, State: Running, Role: LEADER
I20260926 02:07:44.601591 26593 consensus_queue.cc:237] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 } } peers { permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 } } peers { permanent_uuid: "17e22f81cc54435fa3db75d74f44e284" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 42645 } }
I20260926 02:07:44.602480 26594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.602027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.130:48649)
0926 02:07:44.602193 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.602319 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 02:07:44.605218 25829 catalog_manager.cc:5495] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17e22f81cc54435fa3db75d74f44e284 (127.18.159.130). New cstate: current_term: 1 leader_uuid: "17e22f81cc54435fa3db75d74f44e284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17e22f81cc54435fa3db75d74f44e284" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 42645 } health_report { overall_health: HEALTHY } } }
I20260926 02:07:44.648288 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.642864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57144 (local address 127.18.159.130:42645)
0926 02:07:44.643064 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:44.643069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:44.643089 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:07:44.643148 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:44.643152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:44.643217 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:44.643297 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:07:44.643819 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.644958 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.646374 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.646772 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.646896 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:44.647343 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:07:44.647468 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:07:44.647551 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:44.647760 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 02:07:44.647910 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:07:44.651851 26160 raft_consensus.cc:1261] T 46e25c919f9d41e28d7a3523f0477495 P 3f30c24fe8b7457ba6f65e5ef22fee66 [term 1 FOLLOWER]: Refusing update from remote peer 17e22f81cc54435fa3db75d74f44e284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:44.651852 26441 raft_consensus.cc:1261] T 46e25c919f9d41e28d7a3523f0477495 P f9a809af0ebb4c5b8b485259aeb2fb1c [term 1 FOLLOWER]: Refusing update from remote peer 17e22f81cc54435fa3db75d74f44e284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:44.652586 26593 consensus_queue.cc:1035] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a809af0ebb4c5b8b485259aeb2fb1c" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:44.652930 26595 consensus_queue.cc:1035] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f30c24fe8b7457ba6f65e5ef22fee66" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 34533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:44.655652 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.655047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.129:34155)
0926 02:07:44.655312 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.655431 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260926 02:07:44.657642 26590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.657135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.131:41741)
0926 02:07:44.657357 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.657453 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260926 02:07:44.664744 26590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.664257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.131:34765)
0926 02:07:44.664455 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.664553 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 02:07:44.671626 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.670978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.129:59127)
0926 02:07:44.671216 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.671393 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 02:07:44.673287 26594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.672803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.130:57873)
0926 02:07:44.673011 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.673100 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260926 02:07:44.674439 26594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.673960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.18.159.130:44773)
0926 02:07:44.674156 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.674251 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 02:07:44.679713 25829 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:44.680024 25829 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:44.686951 25829 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d7e2176af0a475eab3956ac474f6320: Sending DeleteTablet for 3 replicas of tablet 46e25c919f9d41e28d7a3523f0477495
I20260926 02:07:44.687937 26421 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e25c919f9d41e28d7a3523f0477495 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:44 UTC) from {username='oryx', principal='oryx/127.18.159.189@KRBTEST.COM'} at 127.0.0.1:47848
I20260926 02:07:44.688217 26280 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e25c919f9d41e28d7a3523f0477495 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:44 UTC) from {username='oryx', principal='oryx/127.18.159.189@KRBTEST.COM'} at 127.0.0.1:57128
I20260926 02:07:44.688429 26140 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e25c919f9d41e28d7a3523f0477495 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:44 UTC) from {username='oryx', principal='oryx/127.18.159.189@KRBTEST.COM'} at 127.0.0.1:49816
I20260926 02:07:44.688860 26139 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e25c919f9d41e28d7a3523f0477495 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:44 UTC) from {username='oryx', principal='oryx/127.18.159.189@KRBTEST.COM'} at 127.0.0.1:49816
I20260926 02:07:44.689420 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 25995
I20260926 02:07:44.692731 26610 tablet_replica.cc:330] T 46e25c919f9d41e28d7a3523f0477495 P 3f30c24fe8b7457ba6f65e5ef22fee66: stopping tablet replica
I20260926 02:07:44.693027 26610 raft_consensus.cc:2229] T 46e25c919f9d41e28d7a3523f0477495 P 3f30c24fe8b7457ba6f65e5ef22fee66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:44.693423 26610 raft_consensus.cc:2258] T 46e25c919f9d41e28d7a3523f0477495 P 3f30c24fe8b7457ba6f65e5ef22fee66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:44.694069 26608 tablet_replica.cc:330] T 46e25c919f9d41e28d7a3523f0477495 P f9a809af0ebb4c5b8b485259aeb2fb1c: stopping tablet replica
I20260926 02:07:44.694211 26608 raft_consensus.cc:2229] T 46e25c919f9d41e28d7a3523f0477495 P f9a809af0ebb4c5b8b485259aeb2fb1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:44.694422 26608 raft_consensus.cc:2258] T 46e25c919f9d41e28d7a3523f0477495 P f9a809af0ebb4c5b8b485259aeb2fb1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:44.694988 26609 tablet_replica.cc:330] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284: stopping tablet replica
I20260926 02:07:44.695142 26609 raft_consensus.cc:2229] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:07:44.695482 26609 raft_consensus.cc:2258] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:44.696569 26609 ts_tablet_manager.cc:1905] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:44.697541 26608 ts_tablet_manager.cc:1905] T 46e25c919f9d41e28d7a3523f0477495 P f9a809af0ebb4c5b8b485259aeb2fb1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:44.704061 26609 ts_tablet_manager.cc:1918] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:07:44.704281 26609 log.cc:1196] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/46e25c919f9d41e28d7a3523f0477495
I20260926 02:07:44.704785 26609 ts_tablet_manager.cc:1939] T 46e25c919f9d41e28d7a3523f0477495 P 17e22f81cc54435fa3db75d74f44e284: Deleting consensus metadata
I20260926 02:07:44.705657 26610 ts_tablet_manager.cc:1905] T 46e25c919f9d41e28d7a3523f0477495 P 3f30c24fe8b7457ba6f65e5ef22fee66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:44.706144 25815 catalog_manager.cc:4841] TS 17e22f81cc54435fa3db75d74f44e284 (127.18.159.130:42645): tablet 46e25c919f9d41e28d7a3523f0477495 (table test-table [id=3810cf79ca7049438d689039da0b07de]) successfully deleted
I20260926 02:07:44.708653 26608 ts_tablet_manager.cc:1918] T 46e25c919f9d41e28d7a3523f0477495 P f9a809af0ebb4c5b8b485259aeb2fb1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:07:44.708823 26608 log.cc:1196] T 46e25c919f9d41e28d7a3523f0477495 P f9a809af0ebb4c5b8b485259aeb2fb1c: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/46e25c919f9d41e28d7a3523f0477495
I20260926 02:07:44.709259 26608 ts_tablet_manager.cc:1939] T 46e25c919f9d41e28d7a3523f0477495 P f9a809af0ebb4c5b8b485259aeb2fb1c: Deleting consensus metadata
I20260926 02:07:44.710441 25816 catalog_manager.cc:4841] TS f9a809af0ebb4c5b8b485259aeb2fb1c (127.18.159.131:41509): tablet 46e25c919f9d41e28d7a3523f0477495 (table test-table [id=3810cf79ca7049438d689039da0b07de]) successfully deleted
I20260926 02:07:44.713130 26610 ts_tablet_manager.cc:1918] T 46e25c919f9d41e28d7a3523f0477495 P 3f30c24fe8b7457ba6f65e5ef22fee66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:07:44.713367 26610 log.cc:1196] T 46e25c919f9d41e28d7a3523f0477495 P 3f30c24fe8b7457ba6f65e5ef22fee66: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/46e25c919f9d41e28d7a3523f0477495
I20260926 02:07:44.713913 26610 ts_tablet_manager.cc:1939] T 46e25c919f9d41e28d7a3523f0477495 P 3f30c24fe8b7457ba6f65e5ef22fee66: Deleting consensus metadata
I20260926 02:07:44.715240 25817 catalog_manager.cc:4841] TS 3f30c24fe8b7457ba6f65e5ef22fee66 (127.18.159.129:34533): tablet 46e25c919f9d41e28d7a3523f0477495 (table test-table [id=3810cf79ca7049438d689039da0b07de]) successfully deleted
I20260926 02:07:44.715421 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.697093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49824 (local address 127.18.159.129:34533)
0926 02:07:44.697298 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:44.697304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:44.697337 (+ 33us) server_negotiation.cc:407] Connection header received
0926 02:07:44.697503 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:44.697508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:44.697586 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:44.697656 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:44.699688 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.700880 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.702828 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.703532 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.703630 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:44.707387 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:44.707408 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:44.707429 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:44.707475 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:44.711878 (+ 4403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:44.712938 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:44.712949 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:44.712953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:44.713124 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:44.713576 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:44.713585 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:44.713589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:44.714183 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:44.714493 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:44.714790 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 02:07:44.715093 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 02:07:44.715946 25817 catalog_manager.cc:4808] TS 3f30c24fe8b7457ba6f65e5ef22fee66 (127.18.159.129:34533): delete failed for tablet 46e25c919f9d41e28d7a3523f0477495 because the tablet was not found. No further retry: Not found: Tablet not found: 46e25c919f9d41e28d7a3523f0477495
I20260926 02:07:44.716140 26200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:44.810276 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 25995
I20260926 02:07:44.832342 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 26217
I20260926 02:07:44.848729 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.833139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57146 (local address 127.18.159.130:42645)
0926 02:07:44.833319 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:44.833325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:44.833347 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:07:44.833515 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:44.833521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:44.833608 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:44.833717 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:44.835053 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.836191 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.837708 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:44.838365 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:44.838503 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:44.841435 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:44.841456 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:44.841476 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:44.841522 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:44.845945 (+ 4423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:44.846791 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:44.846799 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:44.846801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:44.846946 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:44.847266 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:44.847269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:44.847271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:44.847613 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:44.847843 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:44.848292 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 02:07:44.848481 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:07:44.849421 26340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:44.913654 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.913146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.0.0.1:56130)
0926 02:07:44.913338 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.913441 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 02:07:44.914711 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:44.914335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35357 (local address 127.0.0.1:56132)
0926 02:07:44.914477 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:44.914565 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 02:07:44.993011 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 26217
I20260926 02:07:45.015717 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 26358
I20260926 02:07:45.031709 26614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:45.016954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47852 (local address 127.18.159.131:41509)
0926 02:07:45.017531 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:45.017536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:45.017554 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:07:45.017608 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:45.017611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:45.017670 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:45.017773 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:45.019200 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:45.019944 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:45.021565 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:45.022017 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:45.022168 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:45.025275 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:45.025293 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:45.025310 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:45.025360 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:45.028640 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:45.029446 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:45.029454 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:45.029457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:45.029625 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:45.030111 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:45.030115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:45.030117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:45.030627 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:45.030894 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:45.031229 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 02:07:45.031434 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":373,"threads_started":1}
I20260926 02:07:45.032539 26481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:45.126982 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 26358
I20260926 02:07:45.144404 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 25687
I20260926 02:07:45.145658 25860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:45.268297 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 25687
W20260926 02:07:45.289570 26008 connection.cc:535] server connection from 127.0.0.1:58678 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 02:07:45.289976 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 25783
I20260926 02:07:45.291040 26053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:45.406080 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 25783
2026-09-26T02:07:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10107 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:07:45 dist-test-slave-03rn krb5kdc[26620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:07:45 dist-test-slave-03rn krb5kdc[26620](info): set up 2 sockets
Sep 26 02:07:45 dist-test-slave-03rn krb5kdc[26620](info): commencing operation
krb5kdc: starting...
W20260926 02:07:46.525609 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:07:46 dist-test-slave-03rn krb5kdc[26620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:07:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:07:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.190.keytab.
Entry for principal oryx/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.190.keytab.
I20260926 02:07:46.765523 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:41313
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.190.keytab
--principal=oryx/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:41313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:47.112270 26633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:47.112612 26633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:47.112807 26633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:47.121352 26633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:07:47.121482 26633 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 02:07:47.121559 26633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:47.121659 26633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:47.121762 26633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:07:47.121865 26633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:07:47.132823 26633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:41313
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.190.keytab
--principal=oryx/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:41313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:47.134256 26633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:47.135697 26633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:47.148794 26638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:47.148897 26639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:47.148875 26641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:47.150027 26633 server_base.cc:1034] running on GCE node
I20260926 02:07:47.151005 26633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:47.152732 26633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:47.154016 26633 hybrid_clock.cc:648] HybridClock initialized: now 1790388467153927 us; error 69 us; skew 500 ppm
I20260926 02:07:47.154397 26633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:47 dist-test-slave-03rn krb5kdc[26620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388467, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:47.158030 26633 init.cc:377] Logged in from keytab as oryx/127.18.159.190@KRBTEST.COM (short username oryx)
I20260926 02:07:47.159705 26633 webserver.cc:460] Webserver started at http://127.18.159.190:45359/ using document root <none> and password file <none>
I20260926 02:07:47.160584 26633 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:47.160728 26633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:47.161067 26633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:47.164314 26633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "d5a72b319a884cba96dcda6a05ef26ee"
format_stamp: "Formatted at 2026-09-26 02:07:47 on dist-test-slave-03rn"
server_key: "c5471df4eeede5f5d14e3f2f3797a36a"
server_key_iv: "0b6531793d59566a19d69d1f34510930"
server_key_version: "encryptionkey@0"
I20260926 02:07:47.165179 26633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "d5a72b319a884cba96dcda6a05ef26ee"
format_stamp: "Formatted at 2026-09-26 02:07:47 on dist-test-slave-03rn"
server_key: "c5471df4eeede5f5d14e3f2f3797a36a"
server_key_iv: "0b6531793d59566a19d69d1f34510930"
server_key_version: "encryptionkey@0"
I20260926 02:07:47.170745 26633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:07:47.174654 26648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:47.176258 26633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:07:47.176505 26633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "d5a72b319a884cba96dcda6a05ef26ee"
format_stamp: "Formatted at 2026-09-26 02:07:47 on dist-test-slave-03rn"
server_key: "c5471df4eeede5f5d14e3f2f3797a36a"
server_key_iv: "0b6531793d59566a19d69d1f34510930"
server_key_version: "encryptionkey@0"
I20260926 02:07:47.176760 26633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:47.192626 26633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:47.197233 26633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:47.197556 26633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:47.220256 26633 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:41313
I20260926 02:07:47.220319 26709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:41313 every 8 connection(s)
I20260926 02:07:47.221717 26633 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:07:47.225698 26710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:47.229466 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 26633
I20260926 02:07:47.229692 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:07:47.230528 19070 external_mini_cluster.cc:1428] Setting key ef6d37dec4c7cfdffb6415051dbd8940
I20260926 02:07:47.240682 26710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee: Bootstrap starting.
I20260926 02:07:47.245172 26710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:47.246716 26710 log.cc:824] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:07:47 dist-test-slave-03rn krb5kdc[26620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM
I20260926 02:07:47.249931 26710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee: No bootstrap required, opened a new log
I20260926 02:07:47.256443 26713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:47.232995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.18.159.190:41313)
0926 02:07:47.233887 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:47.233901 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:47.233947 (+ 46us) server_negotiation.cc:407] Connection header received
0926 02:07:47.235728 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:47.235765 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:47.236428 (+ 663us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:47.236944 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:47.238886 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:47.240360 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:47.242240 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:47.243068 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:47.243246 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:47.248310 (+ 5064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:47.248319 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:47.248337 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:47.248381 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:47.251709 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:47.252638 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:47.252646 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:47.252653 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:47.252854 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:47.253307 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:47.253311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:47.253313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:47.253909 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:47.254260 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:47.254769 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 02:07:47.255235 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":424,"threads_started":1}
I20260926 02:07:47.258469 26710 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a72b319a884cba96dcda6a05ef26ee" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 41313 } }
I20260926 02:07:47.258939 26710 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:47.259079 26710 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5a72b319a884cba96dcda6a05ef26ee, State: Initialized, Role: FOLLOWER
I20260926 02:07:47.259805 26710 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a72b319a884cba96dcda6a05ef26ee" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 41313 } }
I20260926 02:07:47.260133 26710 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:07:47.260324 26710 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:07:47.260528 26710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:47.262760 26710 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a72b319a884cba96dcda6a05ef26ee" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 41313 } }
I20260926 02:07:47.263377 26710 leader_election.cc:302] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5a72b319a884cba96dcda6a05ef26ee; no voters:
I20260926 02:07:47.264036 26710 leader_election.cc:288] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:07:47.264235 26715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:47.265820 26715 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee [term 1 LEADER]: Becoming Leader. State: Replica: d5a72b319a884cba96dcda6a05ef26ee, State: Running, Role: LEADER
I20260926 02:07:47.266433 26715 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a72b319a884cba96dcda6a05ef26ee" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 41313 } }
I20260926 02:07:47.266595 26710 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:07:47.269833 26716 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5a72b319a884cba96dcda6a05ef26ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a72b319a884cba96dcda6a05ef26ee" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 41313 } } }
I20260926 02:07:47.270542 26716 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:47.269948 26717 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5a72b319a884cba96dcda6a05ef26ee. Latest consensus state: current_term: 1 leader_uuid: "d5a72b319a884cba96dcda6a05ef26ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a72b319a884cba96dcda6a05ef26ee" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 41313 } } }
I20260926 02:07:47.271134 26717 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:47.275439 26724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:07:47.280422 26724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:07:47.290786 26724 catalog_manager.cc:1347] Generated new cluster ID: f04b2e1866e4478ca4fd8ef65448a742
I20260926 02:07:47.290916 26724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:07:47.308682 26724 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:07:47.310110 26724 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:07:47.318815 26724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee: Generated new TSK 0
I20260926 02:07:47.319736 26724 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.159.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.159.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.159.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.129.keytab.
Entry for principal oryx/127.18.159.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.129.keytab.
I20260926 02:07:47.384027 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:41313
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.129.keytab
--principal=oryx/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:47.730437 26736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:47.730746 26736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:47.730921 26736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:47.738819 26736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:47.738950 26736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:47.739104 26736 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:07:47.739228 26736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:07:47.750540 26736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.129.keytab
--principal=oryx/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.159.190:41313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:47.751978 26736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:47.753419 26736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:47.767220 26744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:47.767283 26742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:47.767807 26741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:47.768352 26736 server_base.cc:1034] running on GCE node
I20260926 02:07:47.769048 26736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:47.770020 26736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:47.771306 26736 hybrid_clock.cc:648] HybridClock initialized: now 1790388467771233 us; error 32 us; skew 500 ppm
I20260926 02:07:47.771802 26736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:47 dist-test-slave-03rn krb5kdc[26620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388467, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:47.775372 26736 init.cc:377] Logged in from keytab as oryx/127.18.159.129@KRBTEST.COM (short username oryx)
I20260926 02:07:47.776995 26736 webserver.cc:460] Webserver started at http://127.18.159.129:43369/ using document root <none> and password file <none>
I20260926 02:07:47.777681 26736 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:47.777779 26736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:47.778053 26736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:47.781327 26736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "014cf8b75ec445bf97e96dd105efbdae"
format_stamp: "Formatted at 2026-09-26 02:07:47 on dist-test-slave-03rn"
server_key: "5b5df4cd5754babf8e62991b2a3f3ee8"
server_key_iv: "773e671aba608316f13eba4f10ef58f8"
server_key_version: "encryptionkey@0"
I20260926 02:07:47.782184 26736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "014cf8b75ec445bf97e96dd105efbdae"
format_stamp: "Formatted at 2026-09-26 02:07:47 on dist-test-slave-03rn"
server_key: "5b5df4cd5754babf8e62991b2a3f3ee8"
server_key_iv: "773e671aba608316f13eba4f10ef58f8"
server_key_version: "encryptionkey@0"
I20260926 02:07:47.787619 26736 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:07:47.791523 26751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:47.793118 26736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:07:47.793316 26736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "014cf8b75ec445bf97e96dd105efbdae"
format_stamp: "Formatted at 2026-09-26 02:07:47 on dist-test-slave-03rn"
server_key: "5b5df4cd5754babf8e62991b2a3f3ee8"
server_key_iv: "773e671aba608316f13eba4f10ef58f8"
server_key_version: "encryptionkey@0"
I20260926 02:07:47.793558 26736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:47.807435 26736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:47.811950 26736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:47.812300 26736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:47.815289 26736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:47.815384 26736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:47.815532 26736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:47.815657 26736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:07:47 dist-test-slave-03rn krb5kdc[26620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388467, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM
I20260926 02:07:47.843725 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:47.815848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:58837 (local address 127.18.159.190:41313)
0926 02:07:47.816627 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:47.816634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:47.817872 (+ 1238us) server_negotiation.cc:407] Connection header received
0926 02:07:47.819988 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:47.819996 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:47.820114 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:47.820288 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:47.822821 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:47.824147 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:47.827786 (+ 3639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:47.828612 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:47.828780 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:47.834951 (+ 6171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:47.834957 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:47.834960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:47.835000 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:47.838804 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:47.840254 (+ 1450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:47.840264 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:47.840270 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:47.840453 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:47.841132 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:47.841140 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:47.841147 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:47.841428 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:47.841582 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:47.842812 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0926 02:07:47.843144 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":538,"threads_started":1}
I20260926 02:07:47.844434 26761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:47.816708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:41313 (local address 127.18.159.129:58837)
0926 02:07:47.817675 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:47.817727 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:47.819679 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:47.820616 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:47.820627 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:47.821614 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:47.822571 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:47.822609 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:47.824425 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:47.824431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:47.827489 (+ 3058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:47.827538 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:47.829220 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:47.829228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:47.829654 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:47.830682 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:47.830735 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:47.834536 (+ 3801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:47.839054 (+ 4518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:47.839069 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:47.839096 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:47.840036 (+ 940us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:47.840615 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:47.840621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:47.840625 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:47.840996 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:47.841575 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:47.841584 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:47.842238 (+ 654us) client_negotiation.cc:769] Sending connection context
0926 02:07:47.842636 (+ 398us) client_negotiation.cc:240] Negotiation successful
0926 02:07:47.843104 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":391,"threads_started":1}
I20260926 02:07:47.870141 26736 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:35563
I20260926 02:07:47.870254 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:47.850584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:42235 (local address 127.18.159.190:41313)
0926 02:07:47.850810 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:47.850820 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:47.851732 (+ 912us) server_negotiation.cc:407] Connection header received
0926 02:07:47.852179 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:47.852186 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:47.852296 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:47.852397 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:47.854650 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:47.855966 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:47.858490 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:47.859404 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:47.859547 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:47.862749 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:47.862755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:47.862758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:47.862794 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:47.866571 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:47.867734 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:47.867742 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:47.867749 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:47.867934 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:47.868395 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:47.868398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:47.868400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:47.868686 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:47.868887 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:47.869532 (+ 645us) server_negotiation.cc:299] Negotiation successful
0926 02:07:47.869805 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:07:47.870195 26871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:35563 every 8 connection(s)
I20260926 02:07:47.870530 26833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:47.850523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:41313 (local address 127.18.159.129:42235)
0926 02:07:47.851566 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:47.851609 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:47.851950 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:47.852690 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:47.852699 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:47.853689 (+ 990us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:47.854369 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:47.854388 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:47.856231 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:47.856244 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:47.858227 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:47.858247 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:47.859948 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:47.859957 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:47.860255 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:47.861026 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:47.861059 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:47.862464 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:47.866787 (+ 4323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:47.866796 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:47.866800 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:47.867572 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:47.868081 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:47.868087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:47.868091 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:47.868268 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:47.868862 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:47.868873 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:47.869127 (+ 254us) client_negotiation.cc:769] Sending connection context
0926 02:07:47.869344 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 02:07:47.869814 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"thread_start_us":547,"threads_started":1}
I20260926 02:07:47.871805 26736 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:07:47.874728 26872 heartbeater.cc:347] Connected to a master server at 127.18.159.190:41313
I20260926 02:07:47.874992 26872 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:47.875667 26872 heartbeater.cc:511] Master 127.18.159.190:41313 requested a full tablet report, sending...
I20260926 02:07:47.877720 26665 ts_manager.cc:194] Registered new tserver with Master: 014cf8b75ec445bf97e96dd105efbdae (127.18.159.129:35563)
I20260926 02:07:47.877820 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 26736
I20260926 02:07:47.878018 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:07:47.878656 19070 external_mini_cluster.cc:1428] Setting key 7177dee77d7e9095a448b331001514c2
I20260926 02:07:47.879443 26665 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:58837
WARNING: no policy specified for oryx/127.18.159.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.159.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.159.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.130.keytab.
Entry for principal oryx/127.18.159.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.130.keytab.
I20260926 02:07:47.927831 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:41313
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.130.keytab
--principal=oryx/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:48.273770 26876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:48.274085 26876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:48.274259 26876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:48.282228 26876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:48.282363 26876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:48.282508 26876 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:07:48.282625 26876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:07:48.293197 26876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.130.keytab
--principal=oryx/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.159.190:41313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:48.294553 26876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:48.295957 26876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:48.311126 26881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:48.311215 26884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:48.311163 26882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:48.312057 26876 server_base.cc:1034] running on GCE node
I20260926 02:07:48.312875 26876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:48.313998 26876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:48.315301 26876 hybrid_clock.cc:648] HybridClock initialized: now 1790388468315224 us; error 59 us; skew 500 ppm
I20260926 02:07:48.315676 26876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:48 dist-test-slave-03rn krb5kdc[26620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388468, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:48.319552 26876 init.cc:377] Logged in from keytab as oryx/127.18.159.130@KRBTEST.COM (short username oryx)
I20260926 02:07:48.321271 26876 webserver.cc:460] Webserver started at http://127.18.159.130:39611/ using document root <none> and password file <none>
I20260926 02:07:48.322019 26876 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:48.322158 26876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:48.322489 26876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:48.325613 26876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "b64399a4bad64950a86850c64d1f584e"
format_stamp: "Formatted at 2026-09-26 02:07:48 on dist-test-slave-03rn"
server_key: "617b60bf687ef2a9f1ba93d40f4cd587"
server_key_iv: "dee948f2fecdf6414933d1d95e8f9333"
server_key_version: "encryptionkey@0"
I20260926 02:07:48.326475 26876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "b64399a4bad64950a86850c64d1f584e"
format_stamp: "Formatted at 2026-09-26 02:07:48 on dist-test-slave-03rn"
server_key: "617b60bf687ef2a9f1ba93d40f4cd587"
server_key_iv: "dee948f2fecdf6414933d1d95e8f9333"
server_key_version: "encryptionkey@0"
I20260926 02:07:48.331991 26876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:07:48.335808 26891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:48.337507 26876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:07:48.337724 26876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "b64399a4bad64950a86850c64d1f584e"
format_stamp: "Formatted at 2026-09-26 02:07:48 on dist-test-slave-03rn"
server_key: "617b60bf687ef2a9f1ba93d40f4cd587"
server_key_iv: "dee948f2fecdf6414933d1d95e8f9333"
server_key_version: "encryptionkey@0"
I20260926 02:07:48.337981 26876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:48.351851 26876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:48.356673 26876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:48.356993 26876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:48.359741 26876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:48.359874 26876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:48.360023 26876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:48.360105 26876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:07:48 dist-test-slave-03rn krb5kdc[26620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388468, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM
I20260926 02:07:48.389945 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:48.360714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:53737 (local address 127.18.159.190:41313)
0926 02:07:48.360910 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:48.360917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:48.363000 (+ 2083us) server_negotiation.cc:407] Connection header received
0926 02:07:48.365995 (+ 2995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:48.366003 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:48.366113 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:48.366193 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:48.368914 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:48.370918 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:48.374855 (+ 3937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:48.375346 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:48.375514 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:48.381097 (+ 5583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:48.381107 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:48.381115 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:48.381175 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:48.385376 (+ 4201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:48.386593 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:48.386600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:48.386604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:48.386768 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:48.387473 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:48.387483 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:48.387489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:48.387955 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:48.388110 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:48.389503 (+ 1393us) server_negotiation.cc:299] Negotiation successful
0926 02:07:48.389686 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:07:48.392326 26900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:48.361485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:41313 (local address 127.18.159.130:53737)
0926 02:07:48.362774 (+ 1289us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:48.362834 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:48.365580 (+ 2746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:48.366788 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:48.366801 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:48.367660 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:48.368626 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:48.368656 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:48.371210 (+ 2554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:48.371216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:48.374568 (+ 3352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:48.374601 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:48.375627 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:48.375635 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:48.375882 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:48.376758 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:48.376782 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:48.380711 (+ 3929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:48.385580 (+ 4869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:48.385594 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:48.385621 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:48.386422 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:48.386899 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:48.386904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:48.386908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:48.387277 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:48.388237 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:48.388247 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:48.388959 (+ 712us) client_negotiation.cc:769] Sending connection context
0926 02:07:48.389381 (+ 422us) client_negotiation.cc:240] Negotiation successful
0926 02:07:48.390194 (+ 813us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"thread_start_us":771,"threads_started":1}
I20260926 02:07:48.414415 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:48.398932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:41313 (local address 127.18.159.130:49655)
0926 02:07:48.399597 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:48.399617 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:48.399892 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:48.400372 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:48.400377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:48.401131 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:48.401669 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:48.401682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:48.403069 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:48.403075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:48.404120 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:48.404133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:48.405185 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:48.405193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:48.405365 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:48.406050 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:48.406068 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:48.406836 (+ 768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:48.410607 (+ 3771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:48.410616 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:48.410621 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:48.411363 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:48.411933 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:48.411943 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:48.411947 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:48.412127 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:48.412965 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:48.412974 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:48.413185 (+ 211us) client_negotiation.cc:769] Sending connection context
0926 02:07:48.413367 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 02:07:48.413754 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":381,"threads_started":1}
I20260926 02:07:48.415172 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:48.398813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:49655 (local address 127.18.159.190:41313)
0926 02:07:48.398991 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:48.398996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:48.399848 (+ 852us) server_negotiation.cc:407] Connection header received
0926 02:07:48.400063 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:48.400068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:48.400142 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:48.400285 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:48.401849 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:48.402928 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:48.404304 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:48.404936 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:48.405101 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:48.407031 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:48.407038 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:48.407042 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:48.407091 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:48.410449 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:48.411526 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:48.411536 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:48.411543 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:48.411725 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:48.412286 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:48.412293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:48.412298 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:48.412838 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:48.413000 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:48.414768 (+ 1768us) server_negotiation.cc:299] Negotiation successful
0926 02:07:48.414966 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:07:48.419849 26876 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:39201
I20260926 02:07:48.419874 27010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:39201 every 8 connection(s)
I20260926 02:07:48.421185 26876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:07:48.421448 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 26876
I20260926 02:07:48.421610 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:07:48.422252 19070 external_mini_cluster.cc:1428] Setting key 4b514a954254d883db90b9fe2566ffad
I20260926 02:07:48.423496 27011 heartbeater.cc:347] Connected to a master server at 127.18.159.190:41313
I20260926 02:07:48.423748 27011 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:48.424660 27011 heartbeater.cc:511] Master 127.18.159.190:41313 requested a full tablet report, sending...
I20260926 02:07:48.426110 26665 ts_manager.cc:194] Registered new tserver with Master: b64399a4bad64950a86850c64d1f584e (127.18.159.130:39201)
I20260926 02:07:48.426932 26665 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:53737
WARNING: no policy specified for oryx/127.18.159.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.159.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.159.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.131.keytab.
Entry for principal oryx/127.18.159.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.131.keytab.
I20260926 02:07:48.475893 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:41313
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.131.keytab
--principal=oryx/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:48.827237 27015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:48.827596 27015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:48.827775 27015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:48.835319 27015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:48.835515 27015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:48.835685 27015 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:07:48.835836 27015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:07:48.846673 27015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.131.keytab
--principal=oryx/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.159.190:41313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:48.848145 27015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:48.849651 27015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:48.865034 27020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:48.864996 27023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:48.864930 27021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:48.865959 27015 server_base.cc:1034] running on GCE node
I20260926 02:07:48.866700 27015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:48.867856 27015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:48.869154 27015 hybrid_clock.cc:648] HybridClock initialized: now 1790388468869055 us; error 59 us; skew 500 ppm
I20260926 02:07:48.869590 27015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:48 dist-test-slave-03rn krb5kdc[26620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388468, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:48.873346 27015 init.cc:377] Logged in from keytab as oryx/127.18.159.131@KRBTEST.COM (short username oryx)
I20260926 02:07:48.875159 27015 webserver.cc:460] Webserver started at http://127.18.159.131:33733/ using document root <none> and password file <none>
I20260926 02:07:48.876003 27015 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:48.876123 27015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:48.876529 27015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:48.879688 27015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "847b8494647249fa952fde29bed0276f"
format_stamp: "Formatted at 2026-09-26 02:07:48 on dist-test-slave-03rn"
server_key: "7a434ed497e8c182507c06efbb95fa86"
server_key_iv: "7d04b2761d8e43b4142978fe384bb55a"
server_key_version: "encryptionkey@0"
I20260926 02:07:48.880635 27015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "847b8494647249fa952fde29bed0276f"
format_stamp: "Formatted at 2026-09-26 02:07:48 on dist-test-slave-03rn"
server_key: "7a434ed497e8c182507c06efbb95fa86"
server_key_iv: "7d04b2761d8e43b4142978fe384bb55a"
server_key_version: "encryptionkey@0"
I20260926 02:07:48.882887 26872 heartbeater.cc:503] Master 127.18.159.190:41313 was elected leader, sending a full tablet report...
I20260926 02:07:48.887101 27015 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:07:48.891265 27030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:48.893024 27015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:07:48.893244 27015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "847b8494647249fa952fde29bed0276f"
format_stamp: "Formatted at 2026-09-26 02:07:48 on dist-test-slave-03rn"
server_key: "7a434ed497e8c182507c06efbb95fa86"
server_key_iv: "7d04b2761d8e43b4142978fe384bb55a"
server_key_version: "encryptionkey@0"
I20260926 02:07:48.893512 27015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:48.906863 27015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:48.911932 27015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:48.912288 27015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:48.915501 27015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:48.915694 27015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 02:07:48.915872 27015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:48.915949 27015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:07:48 dist-test-slave-03rn krb5kdc[26620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388468, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM
I20260926 02:07:48.941180 27038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:48.915793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:38281 (local address 127.18.159.190:41313)
0926 02:07:48.916407 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:48.916415 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:48.917611 (+ 1196us) server_negotiation.cc:407] Connection header received
0926 02:07:48.920025 (+ 2414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:48.920032 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:48.920142 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:48.920314 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:48.923420 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:48.924577 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:48.927332 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:48.927800 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:48.927956 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:48.933764 (+ 5808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:48.933774 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:48.933783 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:48.933835 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:48.937492 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:48.938345 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:48.938351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:48.938354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:48.938499 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:48.939026 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:48.939031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:48.939036 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:48.939348 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:48.939502 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:48.940383 (+ 881us) server_negotiation.cc:299] Negotiation successful
0926 02:07:48.940715 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":320,"threads_started":1}
I20260926 02:07:48.942807 27040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:48.916600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:41313 (local address 127.18.159.131:38281)
0926 02:07:48.917414 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:48.917467 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:48.919625 (+ 2158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:48.920682 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:48.920700 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:48.921740 (+ 1040us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:48.923122 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:48.923159 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:48.924833 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:48.924836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:48.927120 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:48.927136 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:48.928082 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:48.928090 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:48.928416 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:48.929382 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:48.929406 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:48.933404 (+ 3998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:48.937676 (+ 4272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:48.937691 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:48.937712 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:48.938197 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:48.938625 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:48.938631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:48.938635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:48.938897 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:48.939501 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:48.939513 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:48.939941 (+ 428us) client_negotiation.cc:769] Sending connection context
0926 02:07:48.940281 (+ 340us) client_negotiation.cc:240] Negotiation successful
0926 02:07:48.941062 (+ 781us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":343,"threads_started":1}
I20260926 02:07:48.969640 27113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:48.951651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:41313 (local address 127.18.159.131:42037)
0926 02:07:48.952504 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:48.952521 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:48.952703 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:48.953103 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:48.953108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:48.953591 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:48.953883 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:48.953892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:48.955015 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:48.955019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:48.956747 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:48.956762 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:48.958131 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:48.958141 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:48.958375 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:48.959160 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:48.959196 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:48.960259 (+ 1063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:48.965371 (+ 5112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:48.965376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:48.965379 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:48.966002 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:48.966593 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:48.966601 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:48.966606 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:48.966829 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:48.967822 (+ 993us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:48.967832 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:48.968285 (+ 453us) client_negotiation.cc:769] Sending connection context
0926 02:07:48.968570 (+ 285us) client_negotiation.cc:240] Negotiation successful
0926 02:07:48.969095 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":509,"threads_started":1}
I20260926 02:07:48.970817 27038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:48.951550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:42037 (local address 127.18.159.190:41313)
0926 02:07:48.951739 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:48.951743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:48.952625 (+ 882us) server_negotiation.cc:407] Connection header received
0926 02:07:48.952856 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:48.952861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:48.952958 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:48.953073 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:48.954072 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:48.954829 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:48.957043 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:48.957882 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:48.958077 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:48.960529 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:48.960540 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:48.960547 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:48.960594 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:48.965154 (+ 4560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:48.966223 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:48.966227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:48.966230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:48.966441 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:48.967086 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:48.967095 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:48.967102 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:48.967559 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:48.967749 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:48.970072 (+ 2323us) server_negotiation.cc:299] Negotiation successful
0926 02:07:48.970373 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:07:48.971845 27015 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:41459
I20260926 02:07:48.971911 27150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:41459 every 8 connection(s)
I20260926 02:07:48.973551 27015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:07:48.975888 27151 heartbeater.cc:347] Connected to a master server at 127.18.159.190:41313
I20260926 02:07:48.976207 27151 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:48.976892 27151 heartbeater.cc:511] Master 127.18.159.190:41313 requested a full tablet report, sending...
I20260926 02:07:48.978392 26665 ts_manager.cc:194] Registered new tserver with Master: 847b8494647249fa952fde29bed0276f (127.18.159.131:41459)
I20260926 02:07:48.979267 26665 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:38281
I20260926 02:07:48.980387 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 27015
I20260926 02:07:48.980552 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:07:48.981077 19070 external_mini_cluster.cc:1428] Setting key 506964febdc2eba87a562cc591bfd0ac
I20260926 02:07:48.984571 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:07:49 dist-test-slave-03rn krb5kdc[26620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388469, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 02:07:49 dist-test-slave-03rn krb5kdc[26620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:07:49 dist-test-slave-03rn krb5kdc[26620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 02:07:49.039397 27038 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:49.026880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44394 (local address 127.18.159.190:41313)
0926 02:07:49.027133 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:49.027140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:49.027369 (+ 229us) server_negotiation.cc:407] Connection header received
0926 02:07:49.027645 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:49.027650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:49.027729 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:49.027858 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:49.029494 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.030380 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.031944 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.032501 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.032666 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.039032 (+ 6366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44394: BlockingRecv error: recv got EOF from 127.0.0.1:44394 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 26 02:07:49 dist-test-slave-03rn krb5kdc[26620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388469, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM
I20260926 02:07:49.057730 27038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.043196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44402 (local address 127.18.159.190:41313)
0926 02:07:49.043455 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:49.043462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:49.043724 (+ 262us) server_negotiation.cc:407] Connection header received
0926 02:07:49.043954 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:49.043967 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:49.044033 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:49.044145 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:49.045522 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.046375 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.047762 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.048304 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.048446 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.052333 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:49.052339 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:49.052343 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:49.052382 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:49.054896 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:49.055649 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:49.055655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:49.055658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:49.055796 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:49.056284 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:49.056291 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:49.056294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:49.056573 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:49.056855 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:49.057174 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 02:07:49.057402 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:07:49.061983 26665 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:07:49.069555 26665 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:07:49.097514 27174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.086497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36182 (local address 127.18.159.129:35563)
0926 02:07:49.088129 (+ 1632us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:49.088139 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:49.089425 (+ 1286us) server_negotiation.cc:407] Connection header received
0926 02:07:49.089742 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:49.089748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:49.089911 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:49.090112 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:49.090669 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.091618 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.095208 (+ 3590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.096312 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.096528 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.096726 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:49.097028 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 02:07:49.097202 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1462,"thread_start_us":1318,"threads_started":1}
I20260926 02:07:49.097973 27172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.083787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50820 (local address 127.18.159.130:39201)
0926 02:07:49.084797 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:49.084810 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:49.088804 (+ 3994us) server_negotiation.cc:407] Connection header received
0926 02:07:49.088974 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:49.088986 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:49.089260 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:49.089452 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:49.090277 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.091761 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.094937 (+ 3176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.096385 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.096567 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.096741 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:49.097490 (+ 749us) server_negotiation.cc:299] Negotiation successful
0926 02:07:49.097681 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":605,"threads_started":1}
I20260926 02:07:49.098441 27169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.085768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:35563 (local address 127.0.0.1:36182)
0926 02:07:49.089316 (+ 3548us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:49.089342 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:49.089520 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:49.090104 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:49.090108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:49.090135 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:49.090495 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.090504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.091770 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.091775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.095016 (+ 3241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.095025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.096543 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.096549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.096762 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.096820 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 02:07:49.096981 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 02:07:49.097253 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3427,"mutex_wait_us":605,"thread_start_us":2574,"threads_started":1}
I20260926 02:07:49.098917 27170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.083047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:41459 (local address 127.0.0.1:42778)
0926 02:07:49.085526 (+ 2479us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:49.085589 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:49.085813 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:49.087865 (+ 2052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:49.087874 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:49.087908 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:49.088485 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.088506 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.091961 (+ 3455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.091966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.094478 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.094492 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.097471 (+ 2979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.097479 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.098018 (+ 539us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.098197 (+ 179us) client_negotiation.cc:769] Sending connection context
0926 02:07:49.098274 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 02:07:49.098441 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2120,"mutex_wait_us":200,"thread_start_us":5981,"threads_started":1}
I20260926 02:07:49.098964 27173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.083283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:39201 (local address 127.0.0.1:50820)
0926 02:07:49.088683 (+ 5400us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:49.088707 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:49.088826 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:49.089675 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:49.089681 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:49.089705 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:49.090073 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.090089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.091891 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.091901 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.094772 (+ 2871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.094787 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.096891 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.096897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.097051 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.097100 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 02:07:49.097182 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 02:07:49.097316 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5214,"mutex_wait_us":167,"thread_start_us":3737,"threads_started":1}
I20260926 02:07:49.099787 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.083386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42778 (local address 127.18.159.131:41459)
0926 02:07:49.087130 (+ 3744us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:49.087140 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:49.087179 (+ 39us) server_negotiation.cc:407] Connection header received
0926 02:07:49.087282 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:49.087289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:49.087564 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:49.088725 (+ 1161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:49.090012 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.091761 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.094652 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.096385 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.096675 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.099224 (+ 2549us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:49.099298 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 02:07:49.099468 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3565,"thread_start_us":3392,"threads_started":1}
I20260926 02:07:49.102500 26941 tablet_service.cc:1467] Processing CreateTablet for tablet f27f5a6139a64e47a419c9a87910f401 (DEFAULT_TABLE table=test-table [id=4509bcda9fdd424e97a4048e0700b5e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:49.102628 26802 tablet_service.cc:1467] Processing CreateTablet for tablet f27f5a6139a64e47a419c9a87910f401 (DEFAULT_TABLE table=test-table [id=4509bcda9fdd424e97a4048e0700b5e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:49.103164 27081 tablet_service.cc:1467] Processing CreateTablet for tablet f27f5a6139a64e47a419c9a87910f401 (DEFAULT_TABLE table=test-table [id=4509bcda9fdd424e97a4048e0700b5e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:49.103688 26941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f27f5a6139a64e47a419c9a87910f401. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:49.104008 26802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f27f5a6139a64e47a419c9a87910f401. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:49.104485 27081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f27f5a6139a64e47a419c9a87910f401. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:49.125684 27176 tablet_bootstrap.cc:492] T f27f5a6139a64e47a419c9a87910f401 P 014cf8b75ec445bf97e96dd105efbdae: Bootstrap starting.
I20260926 02:07:49.131603 27176 tablet_bootstrap.cc:654] T f27f5a6139a64e47a419c9a87910f401 P 014cf8b75ec445bf97e96dd105efbdae: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:49.133420 27176 log.cc:824] T f27f5a6139a64e47a419c9a87910f401 P 014cf8b75ec445bf97e96dd105efbdae: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:49.133479 27175 tablet_bootstrap.cc:492] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e: Bootstrap starting.
I20260926 02:07:49.133692 27177 tablet_bootstrap.cc:492] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f: Bootstrap starting.
I20260926 02:07:49.136802 27177 tablet_bootstrap.cc:654] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:49.136983 27175 tablet_bootstrap.cc:654] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:49.137151 27176 tablet_bootstrap.cc:492] T f27f5a6139a64e47a419c9a87910f401 P 014cf8b75ec445bf97e96dd105efbdae: No bootstrap required, opened a new log
I20260926 02:07:49.137477 27176 ts_tablet_manager.cc:1397] T f27f5a6139a64e47a419c9a87910f401 P 014cf8b75ec445bf97e96dd105efbdae: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.001s
I20260926 02:07:49.138041 27177 log.cc:824] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:49.138249 27175 log.cc:824] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:49.141554 27175 tablet_bootstrap.cc:492] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e: No bootstrap required, opened a new log
I20260926 02:07:49.141546 27177 tablet_bootstrap.cc:492] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f: No bootstrap required, opened a new log
I20260926 02:07:49.141862 27177 ts_tablet_manager.cc:1397] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 02:07:49.142063 27175 ts_tablet_manager.cc:1397] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 02:07:49.148569 27176 raft_consensus.cc:348] T f27f5a6139a64e47a419c9a87910f401 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } }
I20260926 02:07:49.149089 27176 raft_consensus.cc:374] T f27f5a6139a64e47a419c9a87910f401 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:49.149188 27176 raft_consensus.cc:729] T f27f5a6139a64e47a419c9a87910f401 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014cf8b75ec445bf97e96dd105efbdae, State: Initialized, Role: FOLLOWER
I20260926 02:07:49.149909 27176 consensus_queue.cc:260] T f27f5a6139a64e47a419c9a87910f401 P 014cf8b75ec445bf97e96dd105efbdae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } }
I20260926 02:07:49.150238 27175 raft_consensus.cc:348] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } }
I20260926 02:07:49.150734 27175 raft_consensus.cc:374] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:49.150844 27175 raft_consensus.cc:729] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64399a4bad64950a86850c64d1f584e, State: Initialized, Role: FOLLOWER
I20260926 02:07:49.151557 27175 consensus_queue.cc:260] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } }
I20260926 02:07:49.151944 27176 ts_tablet_manager.cc:1428] T f27f5a6139a64e47a419c9a87910f401 P 014cf8b75ec445bf97e96dd105efbdae: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
I20260926 02:07:49.152984 27177 raft_consensus.cc:348] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } }
I20260926 02:07:49.153483 27177 raft_consensus.cc:374] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:49.153595 27177 raft_consensus.cc:729] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 847b8494647249fa952fde29bed0276f, State: Initialized, Role: FOLLOWER
I20260926 02:07:49.154312 27177 consensus_queue.cc:260] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } }
I20260926 02:07:49.157560 27151 heartbeater.cc:503] Master 127.18.159.190:41313 was elected leader, sending a full tablet report...
I20260926 02:07:49.157780 27011 heartbeater.cc:503] Master 127.18.159.190:41313 was elected leader, sending a full tablet report...
I20260926 02:07:49.157827 27177 ts_tablet_manager.cc:1428] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20260926 02:07:49.158247 27175 ts_tablet_manager.cc:1428] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e: Time spent starting tablet: real 0.016s user 0.009s sys 0.006s
W20260926 02:07:49.173367 27012 tablet.cc:2369] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:07:49.208954 27182 raft_consensus.cc:482] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:07:49.209261 27182 raft_consensus.cc:504] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } }
I20260926 02:07:49.210987 27182 leader_election.cc:288] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 847b8494647249fa952fde29bed0276f (127.18.159.131:41459), 014cf8b75ec445bf97e96dd105efbdae (127.18.159.129:35563)
I20260926 02:07:49.218133 27174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.211448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:49375 (local address 127.18.159.129:35563)
0926 02:07:49.211732 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:49.211739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:49.212586 (+ 847us) server_negotiation.cc:407] Connection header received
0926 02:07:49.212810 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:49.212817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:49.212883 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:49.212999 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:49.213669 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.214396 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.216327 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.217133 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.217355 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.217431 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:49.217729 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 02:07:49.217858 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260926 02:07:49.218339 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.211480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:56445 (local address 127.18.159.131:41459)
0926 02:07:49.211686 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:49.211693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:49.212766 (+ 1073us) server_negotiation.cc:407] Connection header received
0926 02:07:49.212983 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:49.212987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:49.213046 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:49.213154 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:49.213876 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.214581 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.216567 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.217366 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.217589 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.217662 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:49.217960 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 02:07:49.218090 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:07:49.218472 27185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.211422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:41459 (local address 127.18.159.130:56445)
0926 02:07:49.212641 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:49.212655 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:49.212816 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:49.213177 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:49.213181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:49.213223 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:49.213687 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.213697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.214760 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.214765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.216364 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.216378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.217609 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.217615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.217786 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.217809 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:07:49.217915 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 02:07:49.218060 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1088,"mutex_wait_us":24,"thread_start_us":839,"threads_started":1}
I20260926 02:07:49.218725 27184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.211285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:35563 (local address 127.18.159.130:49375)
0926 02:07:49.212461 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:49.212480 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:49.212666 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:49.213016 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:49.213021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:49.213047 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:49.213509 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.213523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.214538 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.214547 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.216099 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.216115 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.217375 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.217381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.217559 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.217584 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 02:07:49.217701 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 02:07:49.217858 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"mutex_wait_us":23,"spinlock_wait_cycles":19328,"thread_start_us":754,"threads_started":1}
I20260926 02:07:49.219651 27101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f27f5a6139a64e47a419c9a87910f401" candidate_uuid: "b64399a4bad64950a86850c64d1f584e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847b8494647249fa952fde29bed0276f" is_pre_election: true
I20260926 02:07:49.219681 26822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f27f5a6139a64e47a419c9a87910f401" candidate_uuid: "b64399a4bad64950a86850c64d1f584e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014cf8b75ec445bf97e96dd105efbdae" is_pre_election: true
I20260926 02:07:49.220243 27101 raft_consensus.cc:2454] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b64399a4bad64950a86850c64d1f584e in term 0.
I20260926 02:07:49.220242 26822 raft_consensus.cc:2454] T f27f5a6139a64e47a419c9a87910f401 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b64399a4bad64950a86850c64d1f584e in term 0.
I20260926 02:07:49.220968 26895 leader_election.cc:302] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 014cf8b75ec445bf97e96dd105efbdae, 847b8494647249fa952fde29bed0276f, b64399a4bad64950a86850c64d1f584e; no voters:
I20260926 02:07:49.221387 27182 raft_consensus.cc:2781] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:07:49.221549 27182 raft_consensus.cc:482] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:07:49.221679 27182 raft_consensus.cc:3037] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:49.223861 27182 raft_consensus.cc:504] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } }
I20260926 02:07:49.224577 27182 leader_election.cc:288] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e [CANDIDATE]: Term 1 election: Requested vote from peers 847b8494647249fa952fde29bed0276f (127.18.159.131:41459), 014cf8b75ec445bf97e96dd105efbdae (127.18.159.129:35563)
I20260926 02:07:49.225087 27101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f27f5a6139a64e47a419c9a87910f401" candidate_uuid: "b64399a4bad64950a86850c64d1f584e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847b8494647249fa952fde29bed0276f"
I20260926 02:07:49.225142 26822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f27f5a6139a64e47a419c9a87910f401" candidate_uuid: "b64399a4bad64950a86850c64d1f584e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014cf8b75ec445bf97e96dd105efbdae"
I20260926 02:07:49.225325 27101 raft_consensus.cc:3037] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:49.225353 26822 raft_consensus.cc:3037] T f27f5a6139a64e47a419c9a87910f401 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Advancing to term 1
W20260926 02:07:49.225570 27152 tablet.cc:2369] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:07:49.228632 27101 raft_consensus.cc:2454] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b64399a4bad64950a86850c64d1f584e in term 1.
I20260926 02:07:49.228626 26822 raft_consensus.cc:2454] T f27f5a6139a64e47a419c9a87910f401 P 014cf8b75ec445bf97e96dd105efbdae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b64399a4bad64950a86850c64d1f584e in term 1.
I20260926 02:07:49.229099 26895 leader_election.cc:302] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014cf8b75ec445bf97e96dd105efbdae, b64399a4bad64950a86850c64d1f584e; no voters:
I20260926 02:07:49.229475 27182 raft_consensus.cc:2781] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:49.229948 27182 raft_consensus.cc:686] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e [term 1 LEADER]: Becoming Leader. State: Replica: b64399a4bad64950a86850c64d1f584e, State: Running, Role: LEADER
I20260926 02:07:49.230515 27182 consensus_queue.cc:237] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } }
I20260926 02:07:49.235841 26663 catalog_manager.cc:5495] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e reported cstate change: term changed from 0 to 1, leader changed from <none> to b64399a4bad64950a86850c64d1f584e (127.18.159.130). New cstate: current_term: 1 leader_uuid: "b64399a4bad64950a86850c64d1f584e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } health_report { overall_health: UNKNOWN } } }
I20260926 02:07:49.271648 27172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.265697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50834 (local address 127.18.159.130:39201)
0926 02:07:49.265924 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:49.265935 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:49.265954 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:07:49.266194 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:49.266197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:49.266293 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:49.266671 (+ 378us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:07:49.267060 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.267850 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.269288 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.269702 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.269808 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.270292 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:07:49.271008 (+ 716us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:07:49.271125 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:49.271246 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 02:07:49.271320 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:07:49.280845 26822 raft_consensus.cc:1261] T f27f5a6139a64e47a419c9a87910f401 P 014cf8b75ec445bf97e96dd105efbdae [term 1 FOLLOWER]: Refusing update from remote peer b64399a4bad64950a86850c64d1f584e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:49.280887 27101 raft_consensus.cc:1261] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f [term 1 FOLLOWER]: Refusing update from remote peer b64399a4bad64950a86850c64d1f584e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:49.282029 27182 consensus_queue.cc:1035] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e [LEADER]: Connected to new peer: Peer: permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:49.282380 27186 consensus_queue.cc:1035] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e [LEADER]: Connected to new peer: Peer: permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:49.320456 26665 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:49.320758 26665 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:49.324342 26665 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee: Sending DeleteTablet for 3 replicas of tablet f27f5a6139a64e47a419c9a87910f401
I20260926 02:07:49.325548 27081 tablet_service.cc:1514] Processing DeleteTablet for tablet f27f5a6139a64e47a419c9a87910f401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:49 UTC) from {username='oryx', principal='oryx/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:42778
I20260926 02:07:49.325752 26802 tablet_service.cc:1514] Processing DeleteTablet for tablet f27f5a6139a64e47a419c9a87910f401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:49 UTC) from {username='oryx', principal='oryx/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:36182
I20260926 02:07:49.326824 26941 tablet_service.cc:1514] Processing DeleteTablet for tablet f27f5a6139a64e47a419c9a87910f401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:49 UTC) from {username='oryx', principal='oryx/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:50820
I20260926 02:07:49.326964 26665 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:07:49.327842 26665 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:07:49.338234 26940 tablet_service.cc:1467] Processing CreateTablet for tablet c8e1d63268d44e9a9c799a43d3aa7a45 (DEFAULT_TABLE table=test-table [id=3db0870e423c41368fbaedfeac2b2b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:49.338490 27080 tablet_service.cc:1467] Processing CreateTablet for tablet c8e1d63268d44e9a9c799a43d3aa7a45 (DEFAULT_TABLE table=test-table [id=3db0870e423c41368fbaedfeac2b2b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:49.338927 26940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e1d63268d44e9a9c799a43d3aa7a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:49.339264 27080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e1d63268d44e9a9c799a43d3aa7a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:49.341437 27199 tablet_replica.cc:330] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f: stopping tablet replica
I20260926 02:07:49.341966 27199 raft_consensus.cc:2229] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:49.342540 27199 raft_consensus.cc:2258] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:49.343659 26801 tablet_service.cc:1467] Processing CreateTablet for tablet c8e1d63268d44e9a9c799a43d3aa7a45 (DEFAULT_TABLE table=test-table [id=3db0870e423c41368fbaedfeac2b2b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:49.344337 26801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e1d63268d44e9a9c799a43d3aa7a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:49.345242 27199 ts_tablet_manager.cc:1905] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:49.348083 27177 tablet_bootstrap.cc:492] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f: Bootstrap starting.
I20260926 02:07:49.349906 27177 tablet_bootstrap.cc:654] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:49.353430 27199 ts_tablet_manager.cc:1918] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:07:49.353675 27199 log.cc:1196] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/f27f5a6139a64e47a419c9a87910f401
I20260926 02:07:49.354264 27199 ts_tablet_manager.cc:1939] T f27f5a6139a64e47a419c9a87910f401 P 847b8494647249fa952fde29bed0276f: Deleting consensus metadata
I20260926 02:07:49.355353 27201 tablet_replica.cc:330] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e: stopping tablet replica
I20260926 02:07:49.355938 27201 raft_consensus.cc:2229] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:07:49.356518 26649 catalog_manager.cc:4841] TS 847b8494647249fa952fde29bed0276f (127.18.159.131:41459): tablet f27f5a6139a64e47a419c9a87910f401 (table test-table [id=4509bcda9fdd424e97a4048e0700b5e8]) successfully deleted
I20260926 02:07:49.357352 27201 raft_consensus.cc:2258] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:49.356612 27200 tablet_replica.cc:330] T f27f5a6139a64e47a419c9a87910f401 P 014cf8b75ec445bf97e96dd105efbdae: stopping tablet replica
I20260926 02:07:49.358001 27200 raft_consensus.cc:2229] T f27f5a6139a64e47a419c9a87910f401 P 014cf8b75ec445bf97e96dd105efbdae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:49.358521 27200 raft_consensus.cc:2258] T f27f5a6139a64e47a419c9a87910f401 P 014cf8b75ec445bf97e96dd105efbdae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:49.360945 27177 tablet_bootstrap.cc:492] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f: No bootstrap required, opened a new log
I20260926 02:07:49.361114 27177 ts_tablet_manager.cc:1397] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260926 02:07:49.362064 27200 ts_tablet_manager.cc:1905] T f27f5a6139a64e47a419c9a87910f401 P 014cf8b75ec445bf97e96dd105efbdae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:49.362071 27201 ts_tablet_manager.cc:1905] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:49.361963 27177 raft_consensus.cc:348] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:49.362190 27177 raft_consensus.cc:374] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:49.362298 27177 raft_consensus.cc:729] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 847b8494647249fa952fde29bed0276f, State: Initialized, Role: FOLLOWER
I20260926 02:07:49.362649 27177 consensus_queue.cc:260] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:49.362990 27175 tablet_bootstrap.cc:492] T c8e1d63268d44e9a9c799a43d3aa7a45 P b64399a4bad64950a86850c64d1f584e: Bootstrap starting.
I20260926 02:07:49.363236 27177 ts_tablet_manager.cc:1428] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:07:49.364959 27175 tablet_bootstrap.cc:654] T c8e1d63268d44e9a9c799a43d3aa7a45 P b64399a4bad64950a86850c64d1f584e: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:49.369632 27176 tablet_bootstrap.cc:492] T c8e1d63268d44e9a9c799a43d3aa7a45 P 014cf8b75ec445bf97e96dd105efbdae: Bootstrap starting.
I20260926 02:07:49.369726 27201 ts_tablet_manager.cc:1918] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:07:49.369915 27201 log.cc:1196] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/f27f5a6139a64e47a419c9a87910f401
I20260926 02:07:49.370502 27201 ts_tablet_manager.cc:1939] T f27f5a6139a64e47a419c9a87910f401 P b64399a4bad64950a86850c64d1f584e: Deleting consensus metadata
I20260926 02:07:49.371456 27176 tablet_bootstrap.cc:654] T c8e1d63268d44e9a9c799a43d3aa7a45 P 014cf8b75ec445bf97e96dd105efbdae: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:49.372553 26651 catalog_manager.cc:4841] TS b64399a4bad64950a86850c64d1f584e (127.18.159.130:39201): tablet f27f5a6139a64e47a419c9a87910f401 (table test-table [id=4509bcda9fdd424e97a4048e0700b5e8]) successfully deleted
I20260926 02:07:49.374178 27176 tablet_bootstrap.cc:492] T c8e1d63268d44e9a9c799a43d3aa7a45 P 014cf8b75ec445bf97e96dd105efbdae: No bootstrap required, opened a new log
I20260926 02:07:49.374339 27176 ts_tablet_manager.cc:1397] T c8e1d63268d44e9a9c799a43d3aa7a45 P 014cf8b75ec445bf97e96dd105efbdae: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260926 02:07:49.375309 27200 ts_tablet_manager.cc:1918] T f27f5a6139a64e47a419c9a87910f401 P 014cf8b75ec445bf97e96dd105efbdae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:07:49.375191 27176 raft_consensus.cc:348] T c8e1d63268d44e9a9c799a43d3aa7a45 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:49.375504 27200 log.cc:1196] T f27f5a6139a64e47a419c9a87910f401 P 014cf8b75ec445bf97e96dd105efbdae: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/f27f5a6139a64e47a419c9a87910f401
I20260926 02:07:49.375602 27175 tablet_bootstrap.cc:492] T c8e1d63268d44e9a9c799a43d3aa7a45 P b64399a4bad64950a86850c64d1f584e: No bootstrap required, opened a new log
I20260926 02:07:49.375631 27176 raft_consensus.cc:374] T c8e1d63268d44e9a9c799a43d3aa7a45 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:49.375782 27175 ts_tablet_manager.cc:1397] T c8e1d63268d44e9a9c799a43d3aa7a45 P b64399a4bad64950a86850c64d1f584e: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.008s
I20260926 02:07:49.375880 27176 raft_consensus.cc:729] T c8e1d63268d44e9a9c799a43d3aa7a45 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014cf8b75ec445bf97e96dd105efbdae, State: Initialized, Role: FOLLOWER
I20260926 02:07:49.376236 27200 ts_tablet_manager.cc:1939] T f27f5a6139a64e47a419c9a87910f401 P 014cf8b75ec445bf97e96dd105efbdae: Deleting consensus metadata
I20260926 02:07:49.376389 27176 consensus_queue.cc:260] T c8e1d63268d44e9a9c799a43d3aa7a45 P 014cf8b75ec445bf97e96dd105efbdae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:49.376782 27175 raft_consensus.cc:348] T c8e1d63268d44e9a9c799a43d3aa7a45 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:49.377077 27175 raft_consensus.cc:374] T c8e1d63268d44e9a9c799a43d3aa7a45 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:49.377185 27175 raft_consensus.cc:729] T c8e1d63268d44e9a9c799a43d3aa7a45 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64399a4bad64950a86850c64d1f584e, State: Initialized, Role: FOLLOWER
I20260926 02:07:49.377584 27176 ts_tablet_manager.cc:1428] T c8e1d63268d44e9a9c799a43d3aa7a45 P 014cf8b75ec445bf97e96dd105efbdae: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 02:07:49.377473 27175 consensus_queue.cc:260] T c8e1d63268d44e9a9c799a43d3aa7a45 P b64399a4bad64950a86850c64d1f584e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:49.378070 27175 ts_tablet_manager.cc:1428] T c8e1d63268d44e9a9c799a43d3aa7a45 P b64399a4bad64950a86850c64d1f584e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:07:49.379989 26652 catalog_manager.cc:4841] TS 014cf8b75ec445bf97e96dd105efbdae (127.18.159.129:35563): tablet f27f5a6139a64e47a419c9a87910f401 (table test-table [id=4509bcda9fdd424e97a4048e0700b5e8]) successfully deleted
I20260926 02:07:49.469156 27183 raft_consensus.cc:482] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:07:49.469399 27183 raft_consensus.cc:504] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:49.471256 27183 leader_election.cc:288] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 014cf8b75ec445bf97e96dd105efbdae (127.18.159.129:35563), b64399a4bad64950a86850c64d1f584e (127.18.159.130:39201)
I20260926 02:07:49.479537 27174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.471794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:58187 (local address 127.18.159.129:35563)
0926 02:07:49.471977 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:49.471986 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:49.472859 (+ 873us) server_negotiation.cc:407] Connection header received
0926 02:07:49.473173 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:49.473183 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:49.473270 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:49.473363 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:49.474313 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.475400 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.477307 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.478480 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.478666 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.478764 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:49.479096 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 02:07:49.479249 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:07:49.479868 27172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.472214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:44751 (local address 127.18.159.130:39201)
0926 02:07:49.472392 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:49.472398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:49.473264 (+ 866us) server_negotiation.cc:407] Connection header received
0926 02:07:49.473407 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:49.473413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:49.473503 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:49.473597 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:49.473980 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.475135 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.477631 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.478921 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.479093 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.479168 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:49.479450 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 02:07:49.479581 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:07:49.479858 27205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.471666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:39201 (local address 127.18.159.131:44751)
0926 02:07:49.473143 (+ 1477us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:49.473160 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:49.473279 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:49.473624 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:49.473628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:49.473648 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:49.473802 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.473809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.475284 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.475290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.477452 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.477467 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.479138 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.479145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.479293 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.479314 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 02:07:49.479418 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 02:07:49.479557 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1320,"thread_start_us":854,"threads_started":1}
I20260926 02:07:49.480675 27206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.471549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:35563 (local address 127.18.159.131:58187)
0926 02:07:49.472726 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:49.472746 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:49.472977 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:49.473767 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:49.473772 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:49.473798 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:49.474144 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.474155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.475569 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.475576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.477148 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.477161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.478674 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.478683 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.478898 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.478936 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 02:07:49.480087 (+ 1151us) client_negotiation.cc:240] Negotiation successful
0926 02:07:49.480302 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1062,"thread_start_us":1279,"threads_started":1}
I20260926 02:07:49.480914 26961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e1d63268d44e9a9c799a43d3aa7a45" candidate_uuid: "847b8494647249fa952fde29bed0276f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64399a4bad64950a86850c64d1f584e" is_pre_election: true
I20260926 02:07:49.481432 26822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e1d63268d44e9a9c799a43d3aa7a45" candidate_uuid: "847b8494647249fa952fde29bed0276f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014cf8b75ec445bf97e96dd105efbdae" is_pre_election: true
I20260926 02:07:49.481415 26961 raft_consensus.cc:2454] T c8e1d63268d44e9a9c799a43d3aa7a45 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 847b8494647249fa952fde29bed0276f in term 0.
I20260926 02:07:49.481670 26822 raft_consensus.cc:2454] T c8e1d63268d44e9a9c799a43d3aa7a45 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 847b8494647249fa952fde29bed0276f in term 0.
I20260926 02:07:49.482106 27033 leader_election.cc:302] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 847b8494647249fa952fde29bed0276f, b64399a4bad64950a86850c64d1f584e; no voters:
I20260926 02:07:49.482537 27183 raft_consensus.cc:2781] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:07:49.482668 27183 raft_consensus.cc:482] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:07:49.482770 27183 raft_consensus.cc:3037] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:49.485164 27183 raft_consensus.cc:504] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:49.485915 27183 leader_election.cc:288] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f [CANDIDATE]: Term 1 election: Requested vote from peers 014cf8b75ec445bf97e96dd105efbdae (127.18.159.129:35563), b64399a4bad64950a86850c64d1f584e (127.18.159.130:39201)
I20260926 02:07:49.486495 26822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e1d63268d44e9a9c799a43d3aa7a45" candidate_uuid: "847b8494647249fa952fde29bed0276f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014cf8b75ec445bf97e96dd105efbdae"
I20260926 02:07:49.486745 26822 raft_consensus.cc:3037] T c8e1d63268d44e9a9c799a43d3aa7a45 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:49.486972 26961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e1d63268d44e9a9c799a43d3aa7a45" candidate_uuid: "847b8494647249fa952fde29bed0276f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64399a4bad64950a86850c64d1f584e"
I20260926 02:07:49.487212 26961 raft_consensus.cc:3037] T c8e1d63268d44e9a9c799a43d3aa7a45 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:49.489864 26822 raft_consensus.cc:2454] T c8e1d63268d44e9a9c799a43d3aa7a45 P 014cf8b75ec445bf97e96dd105efbdae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 847b8494647249fa952fde29bed0276f in term 1.
I20260926 02:07:49.490394 27034 leader_election.cc:302] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014cf8b75ec445bf97e96dd105efbdae, 847b8494647249fa952fde29bed0276f; no voters:
I20260926 02:07:49.490581 26961 raft_consensus.cc:2454] T c8e1d63268d44e9a9c799a43d3aa7a45 P b64399a4bad64950a86850c64d1f584e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 847b8494647249fa952fde29bed0276f in term 1.
I20260926 02:07:49.490767 27183 raft_consensus.cc:2781] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:49.491276 27183 raft_consensus.cc:686] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f [term 1 LEADER]: Becoming Leader. State: Replica: 847b8494647249fa952fde29bed0276f, State: Running, Role: LEADER
I20260926 02:07:49.491755 27183 consensus_queue.cc:237] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:49.495847 26663 catalog_manager.cc:5495] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f reported cstate change: term changed from 0 to 1, leader changed from <none> to 847b8494647249fa952fde29bed0276f (127.18.159.131). New cstate: current_term: 1 leader_uuid: "847b8494647249fa952fde29bed0276f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } health_report { overall_health: HEALTHY } } }
Sep 26 02:07:49 dist-test-slave-03rn krb5kdc[26620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388467, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.190@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM
I20260926 02:07:49.566967 27038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.550686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39854 (local address 127.18.159.190:41313)
0926 02:07:49.550903 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:49.550910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:49.551563 (+ 653us) server_negotiation.cc:407] Connection header received
0926 02:07:49.551799 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:49.551806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:49.551892 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:49.552006 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:49.553314 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.554462 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.555700 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.556343 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.556467 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.560219 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:49.560234 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:49.560239 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:49.560277 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:49.563751 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:49.564806 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:49.564811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:49.564813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:49.564940 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:49.565350 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:49.565357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:49.565360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:49.565641 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:49.565815 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:49.566241 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 02:07:49.566513 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":74}
I20260926 02:07:49.567021 27214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.550744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:41313 (local address 127.0.0.1:39854)
0926 02:07:49.551420 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:49.551443 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:49.551672 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:49.552050 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:49.552053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:49.552903 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:49.553156 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.553165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.554601 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.554605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.555559 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.555571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.556534 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.556538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.556654 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.557424 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:49.557448 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:49.559960 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:49.563965 (+ 4005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:49.563979 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:49.563986 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:49.564585 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:49.565072 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:49.565078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:49.565087 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:49.565216 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:49.565791 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:49.565799 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:49.565974 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 02:07:49.566144 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 02:07:49.566624 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":190,"thread_start_us":363,"threads_started":1}
I20260926 02:07:49.573283 26663 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:39854:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 02:07:49.574407 26663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:07:49.582062 26801 tablet_service.cc:1467] Processing CreateTablet for tablet a491fb3e00fc447bb63a06b5ca853580 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=91bda4b614484ab2854a34abcc97e405]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:07:49.582052 26940 tablet_service.cc:1467] Processing CreateTablet for tablet a491fb3e00fc447bb63a06b5ca853580 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=91bda4b614484ab2854a34abcc97e405]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:07:49.582643 26801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a491fb3e00fc447bb63a06b5ca853580. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:49.582664 26940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a491fb3e00fc447bb63a06b5ca853580. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:49.587867 27176 tablet_bootstrap.cc:492] T a491fb3e00fc447bb63a06b5ca853580 P 014cf8b75ec445bf97e96dd105efbdae: Bootstrap starting.
I20260926 02:07:49.588822 27080 tablet_service.cc:1467] Processing CreateTablet for tablet a491fb3e00fc447bb63a06b5ca853580 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=91bda4b614484ab2854a34abcc97e405]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:07:49.589380 27080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a491fb3e00fc447bb63a06b5ca853580. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:49.589712 27176 tablet_bootstrap.cc:654] T a491fb3e00fc447bb63a06b5ca853580 P 014cf8b75ec445bf97e96dd105efbdae: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:49.590456 27175 tablet_bootstrap.cc:492] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e: Bootstrap starting.
I20260926 02:07:49.592556 27175 tablet_bootstrap.cc:654] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:49.599388 27177 tablet_bootstrap.cc:492] T a491fb3e00fc447bb63a06b5ca853580 P 847b8494647249fa952fde29bed0276f: Bootstrap starting.
I20260926 02:07:49.601140 27175 tablet_bootstrap.cc:492] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e: No bootstrap required, opened a new log
I20260926 02:07:49.601375 27175 ts_tablet_manager.cc:1397] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20260926 02:07:49.601653 27177 tablet_bootstrap.cc:654] T a491fb3e00fc447bb63a06b5ca853580 P 847b8494647249fa952fde29bed0276f: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:49.603947 27175 raft_consensus.cc:348] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:49.604278 27175 raft_consensus.cc:374] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:49.604379 27175 raft_consensus.cc:729] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64399a4bad64950a86850c64d1f584e, State: Initialized, Role: FOLLOWER
I20260926 02:07:49.604700 27175 consensus_queue.cc:260] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:49.605198 27195 tablet_replica.cc:439] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } } }
I20260926 02:07:49.605597 27195 tablet_replica.cc:442] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:07:49.606846 27175 ts_tablet_manager.cc:1428] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260926 02:07:49.610529 27177 tablet_bootstrap.cc:492] T a491fb3e00fc447bb63a06b5ca853580 P 847b8494647249fa952fde29bed0276f: No bootstrap required, opened a new log
I20260926 02:07:49.610529 27176 tablet_bootstrap.cc:492] T a491fb3e00fc447bb63a06b5ca853580 P 014cf8b75ec445bf97e96dd105efbdae: No bootstrap required, opened a new log
I20260926 02:07:49.610708 27176 ts_tablet_manager.cc:1397] T a491fb3e00fc447bb63a06b5ca853580 P 014cf8b75ec445bf97e96dd105efbdae: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.014s
I20260926 02:07:49.610852 27177 ts_tablet_manager.cc:1397] T a491fb3e00fc447bb63a06b5ca853580 P 847b8494647249fa952fde29bed0276f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260926 02:07:49.611509 27176 raft_consensus.cc:348] T a491fb3e00fc447bb63a06b5ca853580 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:49.611768 27176 raft_consensus.cc:374] T a491fb3e00fc447bb63a06b5ca853580 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:49.611552 27177 raft_consensus.cc:348] T a491fb3e00fc447bb63a06b5ca853580 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:49.611850 27176 raft_consensus.cc:729] T a491fb3e00fc447bb63a06b5ca853580 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014cf8b75ec445bf97e96dd105efbdae, State: Initialized, Role: FOLLOWER
I20260926 02:07:49.611967 27177 raft_consensus.cc:374] T a491fb3e00fc447bb63a06b5ca853580 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:49.612105 27177 raft_consensus.cc:729] T a491fb3e00fc447bb63a06b5ca853580 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 847b8494647249fa952fde29bed0276f, State: Initialized, Role: FOLLOWER
I20260926 02:07:49.612108 27176 consensus_queue.cc:260] T a491fb3e00fc447bb63a06b5ca853580 P 014cf8b75ec445bf97e96dd105efbdae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:49.612468 27177 consensus_queue.cc:260] T a491fb3e00fc447bb63a06b5ca853580 P 847b8494647249fa952fde29bed0276f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:49.612762 27176 ts_tablet_manager.cc:1428] T a491fb3e00fc447bb63a06b5ca853580 P 014cf8b75ec445bf97e96dd105efbdae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:07:49.612614 27181 tablet_replica.cc:439] T a491fb3e00fc447bb63a06b5ca853580 P 014cf8b75ec445bf97e96dd105efbdae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } } }
I20260926 02:07:49.613016 27181 tablet_replica.cc:442] T a491fb3e00fc447bb63a06b5ca853580 P 014cf8b75ec445bf97e96dd105efbdae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:07:49.613186 27177 ts_tablet_manager.cc:1428] T a491fb3e00fc447bb63a06b5ca853580 P 847b8494647249fa952fde29bed0276f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 02:07:49.613552 27183 tablet_replica.cc:439] T a491fb3e00fc447bb63a06b5ca853580 P 847b8494647249fa952fde29bed0276f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } } }
I20260926 02:07:49.613870 27183 tablet_replica.cc:442] T a491fb3e00fc447bb63a06b5ca853580 P 847b8494647249fa952fde29bed0276f: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 02:07:49.626366 26873 tablet.cc:2369] T a491fb3e00fc447bb63a06b5ca853580 P 014cf8b75ec445bf97e96dd105efbdae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:07:49.638115 27195 raft_consensus.cc:482] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:07:49.638391 27195 raft_consensus.cc:504] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:49.639230 27195 leader_election.cc:288] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 014cf8b75ec445bf97e96dd105efbdae (127.18.159.129:35563), 847b8494647249fa952fde29bed0276f (127.18.159.131:41459)
I20260926 02:07:49.639740 26822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a491fb3e00fc447bb63a06b5ca853580" candidate_uuid: "b64399a4bad64950a86850c64d1f584e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014cf8b75ec445bf97e96dd105efbdae" is_pre_election: true
I20260926 02:07:49.639737 27101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a491fb3e00fc447bb63a06b5ca853580" candidate_uuid: "b64399a4bad64950a86850c64d1f584e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847b8494647249fa952fde29bed0276f" is_pre_election: true
I20260926 02:07:49.640061 26822 raft_consensus.cc:2454] T a491fb3e00fc447bb63a06b5ca853580 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b64399a4bad64950a86850c64d1f584e in term 0.
I20260926 02:07:49.640069 27101 raft_consensus.cc:2454] T a491fb3e00fc447bb63a06b5ca853580 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b64399a4bad64950a86850c64d1f584e in term 0.
I20260926 02:07:49.640694 26895 leader_election.cc:302] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 014cf8b75ec445bf97e96dd105efbdae, 847b8494647249fa952fde29bed0276f, b64399a4bad64950a86850c64d1f584e; no voters:
I20260926 02:07:49.641172 27195 raft_consensus.cc:2781] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:07:49.641324 27195 raft_consensus.cc:482] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:07:49.641451 27195 raft_consensus.cc:3037] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:49.644219 27195 raft_consensus.cc:504] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:49.644908 27195 leader_election.cc:288] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e [CANDIDATE]: Term 1 election: Requested vote from peers 014cf8b75ec445bf97e96dd105efbdae (127.18.159.129:35563), 847b8494647249fa952fde29bed0276f (127.18.159.131:41459)
I20260926 02:07:49.645505 27101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a491fb3e00fc447bb63a06b5ca853580" candidate_uuid: "b64399a4bad64950a86850c64d1f584e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847b8494647249fa952fde29bed0276f"
I20260926 02:07:49.645557 26822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a491fb3e00fc447bb63a06b5ca853580" candidate_uuid: "b64399a4bad64950a86850c64d1f584e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014cf8b75ec445bf97e96dd105efbdae"
I20260926 02:07:49.645746 27101 raft_consensus.cc:3037] T a491fb3e00fc447bb63a06b5ca853580 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:49.645817 26822 raft_consensus.cc:3037] T a491fb3e00fc447bb63a06b5ca853580 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:49.648883 27101 raft_consensus.cc:2454] T a491fb3e00fc447bb63a06b5ca853580 P 847b8494647249fa952fde29bed0276f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b64399a4bad64950a86850c64d1f584e in term 1.
I20260926 02:07:49.648865 26822 raft_consensus.cc:2454] T a491fb3e00fc447bb63a06b5ca853580 P 014cf8b75ec445bf97e96dd105efbdae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b64399a4bad64950a86850c64d1f584e in term 1.
I20260926 02:07:49.649394 26892 leader_election.cc:302] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 847b8494647249fa952fde29bed0276f, b64399a4bad64950a86850c64d1f584e; no voters:
I20260926 02:07:49.650169 27195 raft_consensus.cc:2781] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:49.650346 27195 raft_consensus.cc:686] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e [term 1 LEADER]: Becoming Leader. State: Replica: b64399a4bad64950a86850c64d1f584e, State: Running, Role: LEADER
I20260926 02:07:49.650621 27195 consensus_queue.cc:237] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:49.652431 27186 tablet_replica.cc:439] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e: TxnStatusTablet state changed. Reason: New leader b64399a4bad64950a86850c64d1f584e. Latest consensus state: current_term: 1 leader_uuid: "b64399a4bad64950a86850c64d1f584e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } } }
I20260926 02:07:49.652784 27186 tablet_replica.cc:442] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:07:49.654031 27219 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 02:07:49.654285 27219 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 02:07:49.654259 26665 catalog_manager.cc:5495] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e reported cstate change: term changed from 0 to 1, leader changed from <none> to b64399a4bad64950a86850c64d1f584e (127.18.159.130). New cstate: current_term: 1 leader_uuid: "b64399a4bad64950a86850c64d1f584e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } health_report { overall_health: UNKNOWN } } }
I20260926 02:07:49.677282 27172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.671473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39272 (local address 127.18.159.130:39201)
0926 02:07:49.671667 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:49.671671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:49.671689 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:07:49.671988 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:49.671992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:49.672059 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:49.672199 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:07:49.672691 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.673432 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.675391 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.675841 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.675958 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.676617 (+ 659us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:07:49.676725 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:07:49.676792 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:49.676986 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 02:07:49.677057 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:07:49.677476 27214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.671358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:39201 (local address 127.0.0.1:39272)
0926 02:07:49.671587 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:49.671609 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:49.671770 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:49.672287 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:49.672292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:49.672306 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:07:49.672549 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.672562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.673584 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.673592 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.675209 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.675238 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.676007 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.676014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.676305 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.676391 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:07:49.676837 (+ 446us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:07:49.676863 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 02:07:49.676963 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 02:07:49.677155 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 02:07:49.681461 27101 raft_consensus.cc:1261] T a491fb3e00fc447bb63a06b5ca853580 P 847b8494647249fa952fde29bed0276f [term 1 FOLLOWER]: Refusing update from remote peer b64399a4bad64950a86850c64d1f584e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:49.681461 26822 raft_consensus.cc:1261] T a491fb3e00fc447bb63a06b5ca853580 P 014cf8b75ec445bf97e96dd105efbdae [term 1 FOLLOWER]: Refusing update from remote peer b64399a4bad64950a86850c64d1f584e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:49.682165 27186 consensus_queue.cc:1035] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e [LEADER]: Connected to new peer: Peer: permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:49.682411 27195 consensus_queue.cc:1035] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e [LEADER]: Connected to new peer: Peer: permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:49.685086 27181 tablet_replica.cc:439] T a491fb3e00fc447bb63a06b5ca853580 P 014cf8b75ec445bf97e96dd105efbdae: TxnStatusTablet state changed. Reason: New leader b64399a4bad64950a86850c64d1f584e. Latest consensus state: current_term: 1 leader_uuid: "b64399a4bad64950a86850c64d1f584e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } } }
I20260926 02:07:49.685426 27181 tablet_replica.cc:442] T a491fb3e00fc447bb63a06b5ca853580 P 014cf8b75ec445bf97e96dd105efbdae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:07:49.687721 27186 tablet_replica.cc:439] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b64399a4bad64950a86850c64d1f584e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } } }
I20260926 02:07:49.687971 27186 tablet_replica.cc:442] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:07:49.688813 27189 mvcc.cc:205] Tried to move back new op lower bound from 7333431171808808960 to 7333431171691593728. Current Snapshot: MvccSnapshot[applied={T|T < 7333431171808808960 or (T in {7333431171808808960})}]
I20260926 02:07:49.693329 27191 mvcc.cc:205] Tried to move back new op lower bound from 7333431171808808960 to 7333431171691593728. Current Snapshot: MvccSnapshot[applied={T|T < 7333431171808808960}]
I20260926 02:07:49.693805 27181 tablet_replica.cc:439] T a491fb3e00fc447bb63a06b5ca853580 P 014cf8b75ec445bf97e96dd105efbdae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b64399a4bad64950a86850c64d1f584e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } } }
I20260926 02:07:49.694114 27181 tablet_replica.cc:442] T a491fb3e00fc447bb63a06b5ca853580 P 014cf8b75ec445bf97e96dd105efbdae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:07:49.696832 27183 tablet_replica.cc:439] T a491fb3e00fc447bb63a06b5ca853580 P 847b8494647249fa952fde29bed0276f: TxnStatusTablet state changed. Reason: New leader b64399a4bad64950a86850c64d1f584e. Latest consensus state: current_term: 1 leader_uuid: "b64399a4bad64950a86850c64d1f584e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } } }
I20260926 02:07:49.697155 27183 tablet_replica.cc:442] T a491fb3e00fc447bb63a06b5ca853580 P 847b8494647249fa952fde29bed0276f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:07:49.699483 27186 tablet_replica.cc:439] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b64399a4bad64950a86850c64d1f584e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } } }
I20260926 02:07:49.699747 27186 tablet_replica.cc:442] T a491fb3e00fc447bb63a06b5ca853580 P b64399a4bad64950a86850c64d1f584e: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:07:49.701944 27183 tablet_replica.cc:439] T a491fb3e00fc447bb63a06b5ca853580 P 847b8494647249fa952fde29bed0276f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b64399a4bad64950a86850c64d1f584e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } } }
I20260926 02:07:49.702314 27183 tablet_replica.cc:442] T a491fb3e00fc447bb63a06b5ca853580 P 847b8494647249fa952fde29bed0276f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:07:49.707433 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.699869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44942 (local address 127.18.159.131:41459)
0926 02:07:49.700036 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:49.700043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:49.700060 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:07:49.700191 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:49.700197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:49.700314 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:49.700475 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:07:49.701136 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.702339 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.704598 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.705257 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.705395 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.705872 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:07:49.706635 (+ 763us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:07:49.706796 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:49.707011 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 02:07:49.707110 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:07:49.723526 27172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.717761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:41731 (local address 127.18.159.130:39201)
0926 02:07:49.717963 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:49.717968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:49.718502 (+ 534us) server_negotiation.cc:407] Connection header received
0926 02:07:49.718755 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:49.718760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:49.718823 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:49.718959 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:07:49.719408 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.720111 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.721654 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.722124 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.722283 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.722734 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:07:49.722865 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:07:49.722961 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:49.723122 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 02:07:49.723197 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:07:49.723608 27225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.717772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:39201 (local address 127.18.159.131:41731)
0926 02:07:49.718392 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:49.718407 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:49.718580 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:49.718973 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:49.718978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:49.718989 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:07:49.719262 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.719269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.720289 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.720296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.721484 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.721496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.722352 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.722360 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.722491 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.722551 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:07:49.722976 (+ 425us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:07:49.722989 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 02:07:49.723089 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 02:07:49.723232 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":343,"threads_started":1}
I20260926 02:07:49.733783 26822 raft_consensus.cc:1261] T c8e1d63268d44e9a9c799a43d3aa7a45 P 014cf8b75ec445bf97e96dd105efbdae [term 1 FOLLOWER]: Refusing update from remote peer 847b8494647249fa952fde29bed0276f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:49.734125 26961 raft_consensus.cc:1261] T c8e1d63268d44e9a9c799a43d3aa7a45 P b64399a4bad64950a86850c64d1f584e [term 1 FOLLOWER]: Refusing update from remote peer 847b8494647249fa952fde29bed0276f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:49.734894 27207 consensus_queue.cc:1035] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f [LEADER]: Connected to new peer: Peer: permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:49.735275 27183 consensus_queue.cc:1035] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:49.741622 27190 mvcc.cc:205] Tried to move back new op lower bound from 7333431172022542336 to 7333431171042914304. Current Snapshot: MvccSnapshot[applied={T|T < 7333431172022542336}]
I20260926 02:07:49.795231 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.774848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:40347 (local address 127.18.159.131:41459)
0926 02:07:49.775051 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:49.775056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:49.788881 (+ 13825us) server_negotiation.cc:407] Connection header received
0926 02:07:49.789024 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:49.789030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:49.789123 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:49.789225 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:07:49.789907 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.791016 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.793280 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.793701 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.793802 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.794394 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:07:49.794539 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:07:49.794621 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:49.794800 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 02:07:49.794883 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 02:07:49.795568 27231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.774747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:41459 (local address 127.18.159.130:40347)
0926 02:07:49.788740 (+ 13993us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:49.788758 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:49.788897 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:49.789275 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:49.789282 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:49.789294 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:07:49.789712 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.789725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.791186 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.791192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.793092 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:49.793114 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.793888 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.793895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:49.794121 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.794216 (+ 95us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:07:49.794645 (+ 429us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:07:49.794660 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 02:07:49.794768 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 02:07:49.794942 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13795,"thread_start_us":13655,"threads_started":1}
I20260926 02:07:49.834610 26663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:49.834880 26663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:49.838537 26663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee: Sending DeleteTablet for 3 replicas of tablet c8e1d63268d44e9a9c799a43d3aa7a45
I20260926 02:07:49.839491 26801 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e1d63268d44e9a9c799a43d3aa7a45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:49 UTC) from {username='oryx', principal='oryx/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:36182
I20260926 02:07:49.839774 27200 tablet_replica.cc:330] T c8e1d63268d44e9a9c799a43d3aa7a45 P 014cf8b75ec445bf97e96dd105efbdae: stopping tablet replica
I20260926 02:07:49.839715 26940 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e1d63268d44e9a9c799a43d3aa7a45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:49 UTC) from {username='oryx', principal='oryx/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:50820
I20260926 02:07:49.839963 27200 raft_consensus.cc:2229] T c8e1d63268d44e9a9c799a43d3aa7a45 P 014cf8b75ec445bf97e96dd105efbdae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:49.840008 27201 tablet_replica.cc:330] T c8e1d63268d44e9a9c799a43d3aa7a45 P b64399a4bad64950a86850c64d1f584e: stopping tablet replica
I20260926 02:07:49.840142 27200 raft_consensus.cc:2258] T c8e1d63268d44e9a9c799a43d3aa7a45 P 014cf8b75ec445bf97e96dd105efbdae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:49.840226 27201 raft_consensus.cc:2229] T c8e1d63268d44e9a9c799a43d3aa7a45 P b64399a4bad64950a86850c64d1f584e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:49.840448 27201 raft_consensus.cc:2258] T c8e1d63268d44e9a9c799a43d3aa7a45 P b64399a4bad64950a86850c64d1f584e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:49.840868 27080 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e1d63268d44e9a9c799a43d3aa7a45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:49 UTC) from {username='oryx', principal='oryx/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:42778
I20260926 02:07:49.841187 27199 tablet_replica.cc:330] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f: stopping tablet replica
I20260926 02:07:49.841372 27199 raft_consensus.cc:2229] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:07:49.842044 27201 ts_tablet_manager.cc:1905] T c8e1d63268d44e9a9c799a43d3aa7a45 P b64399a4bad64950a86850c64d1f584e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:49.843335 27200 ts_tablet_manager.cc:1905] T c8e1d63268d44e9a9c799a43d3aa7a45 P 014cf8b75ec445bf97e96dd105efbdae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:49.841850 27199 raft_consensus.cc:2258] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:49.845547 27199 ts_tablet_manager.cc:1905] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:49.850402 27201 ts_tablet_manager.cc:1918] T c8e1d63268d44e9a9c799a43d3aa7a45 P b64399a4bad64950a86850c64d1f584e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:07:49.850646 27201 log.cc:1196] T c8e1d63268d44e9a9c799a43d3aa7a45 P b64399a4bad64950a86850c64d1f584e: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/c8e1d63268d44e9a9c799a43d3aa7a45
I20260926 02:07:49.851091 27201 ts_tablet_manager.cc:1939] T c8e1d63268d44e9a9c799a43d3aa7a45 P b64399a4bad64950a86850c64d1f584e: Deleting consensus metadata
I20260926 02:07:49.852366 26651 catalog_manager.cc:4841] TS b64399a4bad64950a86850c64d1f584e (127.18.159.130:39201): tablet c8e1d63268d44e9a9c799a43d3aa7a45 (table test-table [id=3db0870e423c41368fbaedfeac2b2b03]) successfully deleted
I20260926 02:07:49.858740 27200 ts_tablet_manager.cc:1918] T c8e1d63268d44e9a9c799a43d3aa7a45 P 014cf8b75ec445bf97e96dd105efbdae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:07:49.859025 27200 log.cc:1196] T c8e1d63268d44e9a9c799a43d3aa7a45 P 014cf8b75ec445bf97e96dd105efbdae: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/c8e1d63268d44e9a9c799a43d3aa7a45
I20260926 02:07:49.859520 27200 ts_tablet_manager.cc:1939] T c8e1d63268d44e9a9c799a43d3aa7a45 P 014cf8b75ec445bf97e96dd105efbdae: Deleting consensus metadata
I20260926 02:07:49.860782 26652 catalog_manager.cc:4841] TS 014cf8b75ec445bf97e96dd105efbdae (127.18.159.129:35563): tablet c8e1d63268d44e9a9c799a43d3aa7a45 (table test-table [id=3db0870e423c41368fbaedfeac2b2b03]) successfully deleted
I20260926 02:07:49.866657 27199 ts_tablet_manager.cc:1918] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:07:49.866852 27199 log.cc:1196] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/c8e1d63268d44e9a9c799a43d3aa7a45
I20260926 02:07:49.867311 27199 ts_tablet_manager.cc:1939] T c8e1d63268d44e9a9c799a43d3aa7a45 P 847b8494647249fa952fde29bed0276f: Deleting consensus metadata
I20260926 02:07:49.868674 26649 catalog_manager.cc:4841] TS 847b8494647249fa952fde29bed0276f (127.18.159.131:41459): tablet c8e1d63268d44e9a9c799a43d3aa7a45 (table test-table [id=3db0870e423c41368fbaedfeac2b2b03]) successfully deleted
Sep 26 02:07:49 dist-test-slave-03rn krb5kdc[26620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:07:49 dist-test-slave-03rn krb5kdc[26620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:07:49 dist-test-slave-03rn krb5kdc[26620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 02:07:49.933147 27038 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:49.918736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39870 (local address 127.18.159.190:41313)
0926 02:07:49.918924 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:49.918930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:49.919515 (+ 585us) server_negotiation.cc:407] Connection header received
0926 02:07:49.919718 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:49.919725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:49.919816 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:49.919939 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:49.921743 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.922978 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.925463 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.926201 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.926347 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.932864 (+ 6517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39870: BlockingRecv error: recv got EOF from 127.0.0.1:39870 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 02:07:49 dist-test-slave-03rn krb5kdc[26620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM
I20260926 02:07:49.957829 27038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:49.937263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39886 (local address 127.18.159.190:41313)
0926 02:07:49.937463 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:49.937470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:49.937982 (+ 512us) server_negotiation.cc:407] Connection header received
0926 02:07:49.938206 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:49.938212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:49.938306 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:49.938380 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:49.940208 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.941333 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.943472 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:49.944131 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:49.944313 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:49.949657 (+ 5344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:49.949665 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:49.949670 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:49.949723 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:49.953396 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:49.954359 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:49.954365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:49.954369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:49.954543 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:49.955064 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:49.955070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:49.955074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:49.955450 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:49.955701 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:49.956059 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 02:07:49.956330 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 02:07:49.962286 26663 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:07:49.963088 26663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:07:49.972536 26940 tablet_service.cc:1467] Processing CreateTablet for tablet c4efe39d180a49e9bfa84c466c97e213 (DEFAULT_TABLE table=test-table [id=8bf97d91db584ea6b4f1abb70ce57e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:49.973091 26940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4efe39d180a49e9bfa84c466c97e213. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:49.976473 26801 tablet_service.cc:1467] Processing CreateTablet for tablet c4efe39d180a49e9bfa84c466c97e213 (DEFAULT_TABLE table=test-table [id=8bf97d91db584ea6b4f1abb70ce57e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:49.977089 26801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4efe39d180a49e9bfa84c466c97e213. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:49.982069 27080 tablet_service.cc:1467] Processing CreateTablet for tablet c4efe39d180a49e9bfa84c466c97e213 (DEFAULT_TABLE table=test-table [id=8bf97d91db584ea6b4f1abb70ce57e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:49.982594 27080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4efe39d180a49e9bfa84c466c97e213. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:50.001230 27176 tablet_bootstrap.cc:492] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae: Bootstrap starting.
I20260926 02:07:50.003041 27176 tablet_bootstrap.cc:654] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:50.007562 27177 tablet_bootstrap.cc:492] T c4efe39d180a49e9bfa84c466c97e213 P 847b8494647249fa952fde29bed0276f: Bootstrap starting.
I20260926 02:07:50.010053 27177 tablet_bootstrap.cc:654] T c4efe39d180a49e9bfa84c466c97e213 P 847b8494647249fa952fde29bed0276f: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:50.019268 27175 tablet_bootstrap.cc:492] T c4efe39d180a49e9bfa84c466c97e213 P b64399a4bad64950a86850c64d1f584e: Bootstrap starting.
I20260926 02:07:50.021508 27175 tablet_bootstrap.cc:654] T c4efe39d180a49e9bfa84c466c97e213 P b64399a4bad64950a86850c64d1f584e: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:50.039469 27175 tablet_bootstrap.cc:492] T c4efe39d180a49e9bfa84c466c97e213 P b64399a4bad64950a86850c64d1f584e: No bootstrap required, opened a new log
I20260926 02:07:50.039779 27175 ts_tablet_manager.cc:1397] T c4efe39d180a49e9bfa84c466c97e213 P b64399a4bad64950a86850c64d1f584e: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.011s
I20260926 02:07:50.040784 27175 raft_consensus.cc:348] T c4efe39d180a49e9bfa84c466c97e213 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:50.041967 27175 raft_consensus.cc:374] T c4efe39d180a49e9bfa84c466c97e213 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:50.042121 27175 raft_consensus.cc:729] T c4efe39d180a49e9bfa84c466c97e213 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64399a4bad64950a86850c64d1f584e, State: Initialized, Role: FOLLOWER
I20260926 02:07:50.042421 27175 consensus_queue.cc:260] T c4efe39d180a49e9bfa84c466c97e213 P b64399a4bad64950a86850c64d1f584e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:50.043270 27177 tablet_bootstrap.cc:492] T c4efe39d180a49e9bfa84c466c97e213 P 847b8494647249fa952fde29bed0276f: No bootstrap required, opened a new log
I20260926 02:07:50.043356 27175 ts_tablet_manager.cc:1428] T c4efe39d180a49e9bfa84c466c97e213 P b64399a4bad64950a86850c64d1f584e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 02:07:50.043526 27177 ts_tablet_manager.cc:1397] T c4efe39d180a49e9bfa84c466c97e213 P 847b8494647249fa952fde29bed0276f: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.004s
I20260926 02:07:50.044400 27177 raft_consensus.cc:348] T c4efe39d180a49e9bfa84c466c97e213 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:50.044725 27177 raft_consensus.cc:374] T c4efe39d180a49e9bfa84c466c97e213 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:50.044763 27176 tablet_bootstrap.cc:492] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae: No bootstrap required, opened a new log
I20260926 02:07:50.044904 27177 raft_consensus.cc:729] T c4efe39d180a49e9bfa84c466c97e213 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 847b8494647249fa952fde29bed0276f, State: Initialized, Role: FOLLOWER
I20260926 02:07:50.044943 27176 ts_tablet_manager.cc:1397] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.009s
I20260926 02:07:50.045269 27177 consensus_queue.cc:260] T c4efe39d180a49e9bfa84c466c97e213 P 847b8494647249fa952fde29bed0276f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:50.045742 27176 raft_consensus.cc:348] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:50.045967 27177 ts_tablet_manager.cc:1428] T c4efe39d180a49e9bfa84c466c97e213 P 847b8494647249fa952fde29bed0276f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:07:50.046041 27176 raft_consensus.cc:374] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:50.046134 27176 raft_consensus.cc:729] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014cf8b75ec445bf97e96dd105efbdae, State: Initialized, Role: FOLLOWER
I20260926 02:07:50.046360 27176 consensus_queue.cc:260] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:50.047752 27176 ts_tablet_manager.cc:1428] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 02:07:50.085472 27181 raft_consensus.cc:482] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:07:50.085753 27181 raft_consensus.cc:504] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:50.087721 27181 leader_election.cc:288] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b64399a4bad64950a86850c64d1f584e (127.18.159.130:39201), 847b8494647249fa952fde29bed0276f (127.18.159.131:41459)
I20260926 02:07:50.108346 27172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:50.089069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:34317 (local address 127.18.159.130:39201)
0926 02:07:50.089375 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:50.089384 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:50.100536 (+ 11152us) server_negotiation.cc:407] Connection header received
0926 02:07:50.100990 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:50.100997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:50.101071 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:50.101128 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:50.101810 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.102974 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.105709 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.107055 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.107221 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:50.107300 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:50.107849 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 02:07:50.108027 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 02:07:50.109683 27249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:50.087929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:39201 (local address 127.18.159.129:34317)
0926 02:07:50.100407 (+ 12478us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:50.100431 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:50.100832 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:50.101350 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:50.101355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:50.101380 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:50.101622 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:50.101632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.103270 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.103276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:50.105472 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:50.105493 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.107455 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.107461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:50.107658 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:50.107698 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 02:07:50.108886 (+ 1188us) client_negotiation.cc:240] Negotiation successful
0926 02:07:50.109123 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12318,"thread_start_us":12625,"threads_started":1}
I20260926 02:07:50.110901 26961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4efe39d180a49e9bfa84c466c97e213" candidate_uuid: "014cf8b75ec445bf97e96dd105efbdae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64399a4bad64950a86850c64d1f584e" is_pre_election: true
I20260926 02:07:50.111219 26961 raft_consensus.cc:2454] T c4efe39d180a49e9bfa84c466c97e213 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014cf8b75ec445bf97e96dd105efbdae in term 0.
I20260926 02:07:50.111874 26754 leader_election.cc:302] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014cf8b75ec445bf97e96dd105efbdae, b64399a4bad64950a86850c64d1f584e; no voters:
I20260926 02:07:50.112396 27181 raft_consensus.cc:2781] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:07:50.112531 27181 raft_consensus.cc:482] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:07:50.112612 27181 raft_consensus.cc:3037] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:50.116546 27181 raft_consensus.cc:504] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:50.117903 26961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4efe39d180a49e9bfa84c466c97e213" candidate_uuid: "014cf8b75ec445bf97e96dd105efbdae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64399a4bad64950a86850c64d1f584e"
I20260926 02:07:50.118170 26961 raft_consensus.cc:3037] T c4efe39d180a49e9bfa84c466c97e213 P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:50.119323 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:50.089603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:43161 (local address 127.18.159.131:41459)
0926 02:07:50.089926 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:50.089935 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:50.110482 (+ 20547us) server_negotiation.cc:407] Connection header received
0926 02:07:50.110801 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:50.110806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:50.110896 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:50.110968 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:07:50.111840 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.113084 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.116099 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.117636 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.117793 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:50.117886 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:50.118811 (+ 925us) server_negotiation.cc:299] Negotiation successful
0926 02:07:50.118983 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 02:07:50.120316 27249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:50.088345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:41459 (local address 127.18.159.129:43161)
0926 02:07:50.110388 (+ 22043us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:50.110408 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:50.110604 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:50.111296 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:50.111301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:50.111435 (+ 134us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:07:50.111650 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:50.111662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.113409 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.113414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:50.115648 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:50.115666 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.118218 (+ 2552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.118226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:50.118480 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:50.118534 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 02:07:50.119803 (+ 1269us) client_negotiation.cc:240] Negotiation successful
0926 02:07:50.119989 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21927,"threads_started":1}
I20260926 02:07:50.121701 26961 raft_consensus.cc:2454] T c4efe39d180a49e9bfa84c466c97e213 P b64399a4bad64950a86850c64d1f584e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014cf8b75ec445bf97e96dd105efbdae in term 1.
I20260926 02:07:50.122328 26754 leader_election.cc:302] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014cf8b75ec445bf97e96dd105efbdae, b64399a4bad64950a86850c64d1f584e; no voters:
I20260926 02:07:50.122730 27181 leader_election.cc:288] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae [CANDIDATE]: Term 1 election: Requested vote from peers b64399a4bad64950a86850c64d1f584e (127.18.159.130:39201), 847b8494647249fa952fde29bed0276f (127.18.159.131:41459)
I20260926 02:07:50.123062 27181 raft_consensus.cc:2781] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:50.123297 27181 raft_consensus.cc:686] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae [term 1 LEADER]: Becoming Leader. State: Replica: 014cf8b75ec445bf97e96dd105efbdae, State: Running, Role: LEADER
I20260926 02:07:50.123895 27181 consensus_queue.cc:237] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:50.133857 27101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4efe39d180a49e9bfa84c466c97e213" candidate_uuid: "014cf8b75ec445bf97e96dd105efbdae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847b8494647249fa952fde29bed0276f" is_pre_election: true
I20260926 02:07:50.134161 27101 raft_consensus.cc:2454] T c4efe39d180a49e9bfa84c466c97e213 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014cf8b75ec445bf97e96dd105efbdae in term 0.
I20260926 02:07:50.134562 27100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4efe39d180a49e9bfa84c466c97e213" candidate_uuid: "014cf8b75ec445bf97e96dd105efbdae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847b8494647249fa952fde29bed0276f"
I20260926 02:07:50.134775 27100 raft_consensus.cc:3037] T c4efe39d180a49e9bfa84c466c97e213 P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:50.135480 26664 catalog_manager.cc:5495] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae reported cstate change: term changed from 0 to 1, leader changed from <none> to 014cf8b75ec445bf97e96dd105efbdae (127.18.159.129). New cstate: current_term: 1 leader_uuid: "014cf8b75ec445bf97e96dd105efbdae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } health_report { overall_health: UNKNOWN } } }
I20260926 02:07:50.138151 27100 raft_consensus.cc:2454] T c4efe39d180a49e9bfa84c466c97e213 P 847b8494647249fa952fde29bed0276f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014cf8b75ec445bf97e96dd105efbdae in term 1.
I20260926 02:07:50.167250 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:50.158948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38316 (local address 127.18.159.129:35563)
0926 02:07:50.159554 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:50.159561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:50.159585 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:07:50.159686 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:50.159691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:50.159810 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:50.159986 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:07:50.160510 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.161591 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.163621 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.164296 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.164439 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:50.165010 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:07:50.165856 (+ 846us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:07:50.166689 (+ 833us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:50.166761 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 02:07:50.166855 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":326,"threads_started":1}
I20260926 02:07:50.172202 27100 raft_consensus.cc:1261] T c4efe39d180a49e9bfa84c466c97e213 P 847b8494647249fa952fde29bed0276f [term 1 FOLLOWER]: Refusing update from remote peer 014cf8b75ec445bf97e96dd105efbdae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:50.172403 26961 raft_consensus.cc:1261] T c4efe39d180a49e9bfa84c466c97e213 P b64399a4bad64950a86850c64d1f584e [term 1 FOLLOWER]: Refusing update from remote peer 014cf8b75ec445bf97e96dd105efbdae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:50.173096 27251 consensus_queue.cc:1035] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae [LEADER]: Connected to new peer: Peer: permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:50.173660 27181 consensus_queue.cc:1035] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:50.214710 26664 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:50.215023 26664 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:50.219164 26664 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee: Sending DeleteTablet for 3 replicas of tablet c4efe39d180a49e9bfa84c466c97e213
I20260926 02:07:50.220237 26801 tablet_service.cc:1514] Processing DeleteTablet for tablet c4efe39d180a49e9bfa84c466c97e213 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:50 UTC) from {username='oryx', principal='oryx/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:36182
I20260926 02:07:50.220614 27200 tablet_replica.cc:330] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae: stopping tablet replica
I20260926 02:07:50.220794 27200 raft_consensus.cc:2229] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:07:50.221383 27200 raft_consensus.cc:2258] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:50.222875 27200 ts_tablet_manager.cc:1905] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:50.229233 26940 tablet_service.cc:1514] Processing DeleteTablet for tablet c4efe39d180a49e9bfa84c466c97e213 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:50 UTC) from {username='oryx', principal='oryx/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:50820
I20260926 02:07:50.229758 27201 tablet_replica.cc:330] T c4efe39d180a49e9bfa84c466c97e213 P b64399a4bad64950a86850c64d1f584e: stopping tablet replica
I20260926 02:07:50.229979 27201 raft_consensus.cc:2229] T c4efe39d180a49e9bfa84c466c97e213 P b64399a4bad64950a86850c64d1f584e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:50.230273 27201 raft_consensus.cc:2258] T c4efe39d180a49e9bfa84c466c97e213 P b64399a4bad64950a86850c64d1f584e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:50.232052 27201 ts_tablet_manager.cc:1905] T c4efe39d180a49e9bfa84c466c97e213 P b64399a4bad64950a86850c64d1f584e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:50.237186 27080 tablet_service.cc:1514] Processing DeleteTablet for tablet c4efe39d180a49e9bfa84c466c97e213 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:50 UTC) from {username='oryx', principal='oryx/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:42778
I20260926 02:07:50.237569 27199 tablet_replica.cc:330] T c4efe39d180a49e9bfa84c466c97e213 P 847b8494647249fa952fde29bed0276f: stopping tablet replica
I20260926 02:07:50.237769 27199 raft_consensus.cc:2229] T c4efe39d180a49e9bfa84c466c97e213 P 847b8494647249fa952fde29bed0276f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:50.238047 27199 raft_consensus.cc:2258] T c4efe39d180a49e9bfa84c466c97e213 P 847b8494647249fa952fde29bed0276f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:50.239665 27199 ts_tablet_manager.cc:1905] T c4efe39d180a49e9bfa84c466c97e213 P 847b8494647249fa952fde29bed0276f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:50.244783 26663 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:07:50.245622 26663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:07:50.256585 26801 tablet_service.cc:1467] Processing CreateTablet for tablet 939096a500904e089a9a83c17147029f (DEFAULT_TABLE table=test-table [id=33ce4b6962d6475aafd48a71595c3281]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:50.257102 26801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939096a500904e089a9a83c17147029f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:50.257009 26940 tablet_service.cc:1467] Processing CreateTablet for tablet 939096a500904e089a9a83c17147029f (DEFAULT_TABLE table=test-table [id=33ce4b6962d6475aafd48a71595c3281]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:50.257529 26940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939096a500904e089a9a83c17147029f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:50.261650 27080 tablet_service.cc:1467] Processing CreateTablet for tablet 939096a500904e089a9a83c17147029f (DEFAULT_TABLE table=test-table [id=33ce4b6962d6475aafd48a71595c3281]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:07:50.262148 27080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939096a500904e089a9a83c17147029f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:50.264297 27176 tablet_bootstrap.cc:492] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae: Bootstrap starting.
I20260926 02:07:50.266120 27176 tablet_bootstrap.cc:654] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:50.278337 27176 tablet_bootstrap.cc:492] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae: No bootstrap required, opened a new log
I20260926 02:07:50.278554 27176 ts_tablet_manager.cc:1397] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260926 02:07:50.279471 27176 raft_consensus.cc:348] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:50.279755 27176 raft_consensus.cc:374] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:50.279848 27176 raft_consensus.cc:729] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014cf8b75ec445bf97e96dd105efbdae, State: Initialized, Role: FOLLOWER
I20260926 02:07:50.280129 27176 consensus_queue.cc:260] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:50.283581 27176 ts_tablet_manager.cc:1428] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260926 02:07:50.289654 27201 ts_tablet_manager.cc:1918] T c4efe39d180a49e9bfa84c466c97e213 P b64399a4bad64950a86850c64d1f584e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:07:50.289959 27201 log.cc:1196] T c4efe39d180a49e9bfa84c466c97e213 P b64399a4bad64950a86850c64d1f584e: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/c4efe39d180a49e9bfa84c466c97e213
I20260926 02:07:50.290616 27201 ts_tablet_manager.cc:1939] T c4efe39d180a49e9bfa84c466c97e213 P b64399a4bad64950a86850c64d1f584e: Deleting consensus metadata
I20260926 02:07:50.291095 27175 tablet_bootstrap.cc:492] T 939096a500904e089a9a83c17147029f P b64399a4bad64950a86850c64d1f584e: Bootstrap starting.
I20260926 02:07:50.292920 27175 tablet_bootstrap.cc:654] T 939096a500904e089a9a83c17147029f P b64399a4bad64950a86850c64d1f584e: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:50.293263 27200 ts_tablet_manager.cc:1918] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:07:50.293411 27200 log.cc:1196] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/c4efe39d180a49e9bfa84c466c97e213
I20260926 02:07:50.293860 27200 ts_tablet_manager.cc:1939] T c4efe39d180a49e9bfa84c466c97e213 P 014cf8b75ec445bf97e96dd105efbdae: Deleting consensus metadata
I20260926 02:07:50.294430 26651 catalog_manager.cc:4841] TS b64399a4bad64950a86850c64d1f584e (127.18.159.130:39201): tablet c4efe39d180a49e9bfa84c466c97e213 (table test-table [id=8bf97d91db584ea6b4f1abb70ce57e0b]) successfully deleted
I20260926 02:07:50.295291 26652 catalog_manager.cc:4841] TS 014cf8b75ec445bf97e96dd105efbdae (127.18.159.129:35563): tablet c4efe39d180a49e9bfa84c466c97e213 (table test-table [id=8bf97d91db584ea6b4f1abb70ce57e0b]) successfully deleted
I20260926 02:07:50.295799 27177 tablet_bootstrap.cc:492] T 939096a500904e089a9a83c17147029f P 847b8494647249fa952fde29bed0276f: Bootstrap starting.
I20260926 02:07:50.297951 27199 ts_tablet_manager.cc:1918] T c4efe39d180a49e9bfa84c466c97e213 P 847b8494647249fa952fde29bed0276f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:07:50.298120 27199 log.cc:1196] T c4efe39d180a49e9bfa84c466c97e213 P 847b8494647249fa952fde29bed0276f: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/c4efe39d180a49e9bfa84c466c97e213
I20260926 02:07:50.298339 27177 tablet_bootstrap.cc:654] T 939096a500904e089a9a83c17147029f P 847b8494647249fa952fde29bed0276f: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:50.298719 27199 ts_tablet_manager.cc:1939] T c4efe39d180a49e9bfa84c466c97e213 P 847b8494647249fa952fde29bed0276f: Deleting consensus metadata
I20260926 02:07:50.300309 26649 catalog_manager.cc:4841] TS 847b8494647249fa952fde29bed0276f (127.18.159.131:41459): tablet c4efe39d180a49e9bfa84c466c97e213 (table test-table [id=8bf97d91db584ea6b4f1abb70ce57e0b]) successfully deleted
I20260926 02:07:50.305648 27175 tablet_bootstrap.cc:492] T 939096a500904e089a9a83c17147029f P b64399a4bad64950a86850c64d1f584e: No bootstrap required, opened a new log
I20260926 02:07:50.305866 27175 ts_tablet_manager.cc:1397] T 939096a500904e089a9a83c17147029f P b64399a4bad64950a86850c64d1f584e: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260926 02:07:50.306807 27175 raft_consensus.cc:348] T 939096a500904e089a9a83c17147029f P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:50.307080 27175 raft_consensus.cc:374] T 939096a500904e089a9a83c17147029f P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:50.307159 27175 raft_consensus.cc:729] T 939096a500904e089a9a83c17147029f P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64399a4bad64950a86850c64d1f584e, State: Initialized, Role: FOLLOWER
I20260926 02:07:50.307439 27175 consensus_queue.cc:260] T 939096a500904e089a9a83c17147029f P b64399a4bad64950a86850c64d1f584e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:50.308023 27175 ts_tablet_manager.cc:1428] T 939096a500904e089a9a83c17147029f P b64399a4bad64950a86850c64d1f584e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 02:07:50.312647 27251 raft_consensus.cc:482] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:07:50.312863 27251 raft_consensus.cc:504] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:50.313616 27251 leader_election.cc:288] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b64399a4bad64950a86850c64d1f584e (127.18.159.130:39201), 847b8494647249fa952fde29bed0276f (127.18.159.131:41459)
I20260926 02:07:50.314368 27100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939096a500904e089a9a83c17147029f" candidate_uuid: "014cf8b75ec445bf97e96dd105efbdae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847b8494647249fa952fde29bed0276f" is_pre_election: true
I20260926 02:07:50.315096 26961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939096a500904e089a9a83c17147029f" candidate_uuid: "014cf8b75ec445bf97e96dd105efbdae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64399a4bad64950a86850c64d1f584e" is_pre_election: true
I20260926 02:07:50.315358 26961 raft_consensus.cc:2454] T 939096a500904e089a9a83c17147029f P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014cf8b75ec445bf97e96dd105efbdae in term 0.
I20260926 02:07:50.315828 26754 leader_election.cc:302] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014cf8b75ec445bf97e96dd105efbdae, b64399a4bad64950a86850c64d1f584e; no voters:
I20260926 02:07:50.316139 27251 raft_consensus.cc:2781] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:07:50.316270 27251 raft_consensus.cc:482] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:07:50.316393 27251 raft_consensus.cc:3037] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:50.319553 27177 tablet_bootstrap.cc:492] T 939096a500904e089a9a83c17147029f P 847b8494647249fa952fde29bed0276f: No bootstrap required, opened a new log
I20260926 02:07:50.319818 27177 ts_tablet_manager.cc:1397] T 939096a500904e089a9a83c17147029f P 847b8494647249fa952fde29bed0276f: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.007s
W20260926 02:07:50.320053 26752 leader_election.cc:341] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 847b8494647249fa952fde29bed0276f (127.18.159.131:41459): Illegal state: must be running to vote when last-logged opid is not known
I20260926 02:07:50.320752 27251 raft_consensus.cc:504] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:50.321890 26961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939096a500904e089a9a83c17147029f" candidate_uuid: "014cf8b75ec445bf97e96dd105efbdae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64399a4bad64950a86850c64d1f584e"
I20260926 02:07:50.322117 26961 raft_consensus.cc:3037] T 939096a500904e089a9a83c17147029f P b64399a4bad64950a86850c64d1f584e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:50.322511 27177 raft_consensus.cc:348] T 939096a500904e089a9a83c17147029f P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:50.322842 27177 raft_consensus.cc:374] T 939096a500904e089a9a83c17147029f P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:50.322980 27177 raft_consensus.cc:729] T 939096a500904e089a9a83c17147029f P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 847b8494647249fa952fde29bed0276f, State: Initialized, Role: FOLLOWER
I20260926 02:07:50.323357 27177 consensus_queue.cc:260] T 939096a500904e089a9a83c17147029f P 847b8494647249fa952fde29bed0276f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:50.325160 26961 raft_consensus.cc:2454] T 939096a500904e089a9a83c17147029f P b64399a4bad64950a86850c64d1f584e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014cf8b75ec445bf97e96dd105efbdae in term 1.
I20260926 02:07:50.325631 26754 leader_election.cc:302] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014cf8b75ec445bf97e96dd105efbdae, b64399a4bad64950a86850c64d1f584e; no voters:
I20260926 02:07:50.325948 27181 raft_consensus.cc:2781] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:50.326675 27177 ts_tablet_manager.cc:1428] T 939096a500904e089a9a83c17147029f P 847b8494647249fa952fde29bed0276f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260926 02:07:50.327670 27251 leader_election.cc:288] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [CANDIDATE]: Term 1 election: Requested vote from peers b64399a4bad64950a86850c64d1f584e (127.18.159.130:39201), 847b8494647249fa952fde29bed0276f (127.18.159.131:41459)
I20260926 02:07:50.327801 27100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939096a500904e089a9a83c17147029f" candidate_uuid: "014cf8b75ec445bf97e96dd105efbdae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847b8494647249fa952fde29bed0276f"
I20260926 02:07:50.328060 27100 raft_consensus.cc:3037] T 939096a500904e089a9a83c17147029f P 847b8494647249fa952fde29bed0276f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:50.331451 27100 raft_consensus.cc:2454] T 939096a500904e089a9a83c17147029f P 847b8494647249fa952fde29bed0276f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014cf8b75ec445bf97e96dd105efbdae in term 1.
I20260926 02:07:50.332414 27181 raft_consensus.cc:686] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [term 1 LEADER]: Becoming Leader. State: Replica: 014cf8b75ec445bf97e96dd105efbdae, State: Running, Role: LEADER
I20260926 02:07:50.332840 27181 consensus_queue.cc:237] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } }
I20260926 02:07:50.339243 26664 catalog_manager.cc:5495] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae reported cstate change: term changed from 0 to 1, leader changed from <none> to 014cf8b75ec445bf97e96dd105efbdae (127.18.159.129). New cstate: current_term: 1 leader_uuid: "014cf8b75ec445bf97e96dd105efbdae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014cf8b75ec445bf97e96dd105efbdae" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 35563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 } health_report { overall_health: UNKNOWN } } }
I20260926 02:07:50.421191 27172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:50.402729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:45211 (local address 127.18.159.130:39201)
0926 02:07:50.402954 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:50.402960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:50.413523 (+ 10563us) server_negotiation.cc:407] Connection header received
0926 02:07:50.414138 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:50.414146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:50.414236 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:50.414299 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:07:50.415143 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.416344 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.418619 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.419408 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.419508 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:50.420303 (+ 795us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:07:50.420479 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:07:50.420542 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:50.420761 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 02:07:50.420861 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 02:07:50.422262 27267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:50.402210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:39201 (local address 127.18.159.129:45211)
0926 02:07:50.413358 (+ 11148us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:50.413384 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:50.413941 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:50.414509 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:50.414513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:50.414526 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:07:50.414932 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:50.414944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.416549 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.416554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:50.418444 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:50.418460 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.419737 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.419744 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:50.419947 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:50.420040 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:07:50.420620 (+ 580us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:07:50.420631 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 02:07:50.421671 (+ 1040us) client_negotiation.cc:240] Negotiation successful
0926 02:07:50.421848 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10886,"thread_start_us":11329,"threads_started":1}
I20260926 02:07:50.436234 26961 raft_consensus.cc:1261] T 939096a500904e089a9a83c17147029f P b64399a4bad64950a86850c64d1f584e [term 1 FOLLOWER]: Refusing update from remote peer 014cf8b75ec445bf97e96dd105efbdae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:50.437079 27261 consensus_queue.cc:1035] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b64399a4bad64950a86850c64d1f584e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:50.439949 27100 raft_consensus.cc:1261] T 939096a500904e089a9a83c17147029f P 847b8494647249fa952fde29bed0276f [term 1 FOLLOWER]: Refusing update from remote peer 014cf8b75ec445bf97e96dd105efbdae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:07:50.440691 27261 consensus_queue.cc:1035] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [LEADER]: Connected to new peer: Peer: permanent_uuid: "847b8494647249fa952fde29bed0276f" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:07:50.527232 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:50.518731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:46125 (local address 127.18.159.129:35563)
0926 02:07:50.518914 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:50.518920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:50.521339 (+ 2419us) server_negotiation.cc:407] Connection header received
0926 02:07:50.521567 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:50.521571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:50.521641 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:50.521786 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:07:50.522311 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.523257 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.525456 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.525937 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.526091 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:50.526504 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:07:50.526612 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:07:50.526713 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:50.526854 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 02:07:50.526933 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:07:50.527473 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:50.519071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:35563 (local address 127.18.159.130:46125)
0926 02:07:50.521225 (+ 2154us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:50.521247 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:50.521406 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:50.521811 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:50.521815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:50.521826 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:07:50.522127 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:50.522137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.523427 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.523432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:50.525311 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:50.525324 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.526130 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.526138 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:50.526298 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:50.526337 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:07:50.526714 (+ 377us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:07:50.526728 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 02:07:50.526834 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 02:07:50.526972 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1917,"thread_start_us":1768,"threads_started":1}
I20260926 02:07:50.567378 26664 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:50.567641 26664 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:07:50.570909 26664 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5a72b319a884cba96dcda6a05ef26ee: Sending DeleteTablet for 3 replicas of tablet 939096a500904e089a9a83c17147029f
I20260926 02:07:50.571776 26801 tablet_service.cc:1514] Processing DeleteTablet for tablet 939096a500904e089a9a83c17147029f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:50 UTC) from {username='oryx', principal='oryx/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:36182
I20260926 02:07:50.571944 26940 tablet_service.cc:1514] Processing DeleteTablet for tablet 939096a500904e089a9a83c17147029f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:50 UTC) from {username='oryx', principal='oryx/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:50820
I20260926 02:07:50.572111 27200 tablet_replica.cc:330] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae: stopping tablet replica
I20260926 02:07:50.572109 27080 tablet_service.cc:1514] Processing DeleteTablet for tablet 939096a500904e089a9a83c17147029f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:07:50 UTC) from {username='oryx', principal='oryx/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:42778
I20260926 02:07:50.572293 27201 tablet_replica.cc:330] T 939096a500904e089a9a83c17147029f P b64399a4bad64950a86850c64d1f584e: stopping tablet replica
I20260926 02:07:50.572343 27200 raft_consensus.cc:2229] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:07:50.572425 27199 tablet_replica.cc:330] T 939096a500904e089a9a83c17147029f P 847b8494647249fa952fde29bed0276f: stopping tablet replica
I20260926 02:07:50.572495 27201 raft_consensus.cc:2229] T 939096a500904e089a9a83c17147029f P b64399a4bad64950a86850c64d1f584e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:50.572608 27199 raft_consensus.cc:2229] T 939096a500904e089a9a83c17147029f P 847b8494647249fa952fde29bed0276f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:07:50.572669 27200 raft_consensus.cc:2258] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:50.572680 27201 raft_consensus.cc:2258] T 939096a500904e089a9a83c17147029f P b64399a4bad64950a86850c64d1f584e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:50.572788 27199 raft_consensus.cc:2258] T 939096a500904e089a9a83c17147029f P 847b8494647249fa952fde29bed0276f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:07:50.574004 27199 ts_tablet_manager.cc:1905] T 939096a500904e089a9a83c17147029f P 847b8494647249fa952fde29bed0276f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:50.574003 27200 ts_tablet_manager.cc:1905] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:50.574328 27201 ts_tablet_manager.cc:1905] T 939096a500904e089a9a83c17147029f P b64399a4bad64950a86850c64d1f584e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:07:50.581367 27199 ts_tablet_manager.cc:1918] T 939096a500904e089a9a83c17147029f P 847b8494647249fa952fde29bed0276f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:07:50.581554 27199 log.cc:1196] T 939096a500904e089a9a83c17147029f P 847b8494647249fa952fde29bed0276f: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/939096a500904e089a9a83c17147029f
I20260926 02:07:50.581650 27200 ts_tablet_manager.cc:1918] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:07:50.581831 27200 log.cc:1196] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/939096a500904e089a9a83c17147029f
I20260926 02:07:50.581996 27199 ts_tablet_manager.cc:1939] T 939096a500904e089a9a83c17147029f P 847b8494647249fa952fde29bed0276f: Deleting consensus metadata
I20260926 02:07:50.582315 27200 ts_tablet_manager.cc:1939] T 939096a500904e089a9a83c17147029f P 014cf8b75ec445bf97e96dd105efbdae: Deleting consensus metadata
I20260926 02:07:50.582517 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 26736
I20260926 02:07:50.583223 26649 catalog_manager.cc:4841] TS 847b8494647249fa952fde29bed0276f (127.18.159.131:41459): tablet 939096a500904e089a9a83c17147029f (table test-table [id=33ce4b6962d6475aafd48a71595c3281]) successfully deleted
I20260926 02:07:50.583461 26652 catalog_manager.cc:4841] TS 014cf8b75ec445bf97e96dd105efbdae (127.18.159.129:35563): tablet 939096a500904e089a9a83c17147029f (table test-table [id=33ce4b6962d6475aafd48a71595c3281]) successfully deleted
I20260926 02:07:50.589591 27201 ts_tablet_manager.cc:1918] T 939096a500904e089a9a83c17147029f P b64399a4bad64950a86850c64d1f584e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:07:50.589771 27201 log.cc:1196] T 939096a500904e089a9a83c17147029f P b64399a4bad64950a86850c64d1f584e: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/939096a500904e089a9a83c17147029f
I20260926 02:07:50.590179 27201 ts_tablet_manager.cc:1939] T 939096a500904e089a9a83c17147029f P b64399a4bad64950a86850c64d1f584e: Deleting consensus metadata
I20260926 02:07:50.591346 26651 catalog_manager.cc:4841] TS b64399a4bad64950a86850c64d1f584e (127.18.159.130:39201): tablet 939096a500904e089a9a83c17147029f (table test-table [id=33ce4b6962d6475aafd48a71595c3281]) successfully deleted
Sep 26 02:07:50 dist-test-slave-03rn krb5kdc[26620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.159.129@KRBTEST.COM
I20260926 02:07:50.612035 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:50.584031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38332 (local address 127.18.159.129:35563)
0926 02:07:50.584303 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:50.584310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:50.589034 (+ 4724us) server_negotiation.cc:407] Connection header received
0926 02:07:50.589251 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:50.589255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:50.589350 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:50.589485 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:50.590958 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.591897 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.593239 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.593711 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.593867 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:50.605806 (+ 11939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:50.605822 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:50.605837 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:50.605878 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:50.609033 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:50.609837 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:50.609846 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:50.609848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:50.609979 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:50.610399 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:50.610403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:50.610405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:50.610780 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:50.611151 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:50.611429 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 02:07:50.611658 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:07:50.612874 26867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:50.736968 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 26736
I20260926 02:07:50.759337 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 26876
Sep 26 02:07:50 dist-test-slave-03rn krb5kdc[26620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.159.130@KRBTEST.COM
I20260926 02:07:50.775936 27172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:50.760314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39288 (local address 127.18.159.130:39201)
0926 02:07:50.760491 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:50.760497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:50.760517 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:07:50.760652 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:50.760656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:50.760715 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:50.760839 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:50.762079 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.762776 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.764950 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.765372 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.765526 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:50.769852 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:50.769874 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:50.769890 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:50.769931 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:50.773148 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:50.773954 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:50.773962 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:50.773965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:50.774104 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:50.774562 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:50.774566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:50.774569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:50.774967 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:50.775224 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:50.775506 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 02:07:50.775683 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:07:50.776690 27006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:50.920765 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 26876
I20260926 02:07:50.946880 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 27015
Sep 26 02:07:50 dist-test-slave-03rn krb5kdc[26620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.159.131@KRBTEST.COM
I20260926 02:07:50.962769 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:50.947714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44950 (local address 127.18.159.131:41459)
0926 02:07:50.948242 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:50.948246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:50.948268 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:07:50.948326 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:50.948330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:50.948389 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:50.948494 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:50.949823 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.950553 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.952015 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:50.952587 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:50.952702 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:50.957175 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:50.957192 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:50.957208 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:50.957252 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:50.960010 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:50.960913 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:50.960951 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:50.960953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:50.961097 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:50.961466 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:50.961470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:50.961472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:50.961816 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:50.962076 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:50.962331 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 02:07:50.962525 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":275,"threads_started":1}
I20260926 02:07:50.963479 27146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:51.090219 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 27015
I20260926 02:07:51.112131 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 26633
I20260926 02:07:51.113349 26705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:51.226738 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 26633
2026-09-26T02:07:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5820 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:07:51 dist-test-slave-03rn krb5kdc[27281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:07:51 dist-test-slave-03rn krb5kdc[27281](info): set up 2 sockets
Sep 26 02:07:51 dist-test-slave-03rn krb5kdc[27281](info): commencing operation
krb5kdc: starting...
W20260926 02:07:52.333444 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:07:52 dist-test-slave-03rn krb5kdc[27281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:07:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:07:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:07:52.571496 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:39713
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:42419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:39713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:52.920686 27294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:52.921017 27294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:52.921172 27294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:52.928730 27294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:07:52.928857 27294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:52.928942 27294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:52.929028 27294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:07:52.929114 27294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:07:52.939504 27294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:42419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:39713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:39713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:52.941035 27294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:52.942531 27294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:52.955502 27300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:52.955459 27299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:52.955778 27302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:52.956526 27294 server_base.cc:1034] running on GCE node
I20260926 02:07:52.957479 27294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:52.959129 27294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:52.960408 27294 hybrid_clock.cc:648] HybridClock initialized: now 1790388472960349 us; error 55 us; skew 500 ppm
I20260926 02:07:52.960829 27294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T02:07:53Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1806 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:07:53 dist-test-slave-03rn krb5kdc[27308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:07:53 dist-test-slave-03rn krb5kdc[27308](info): set up 2 sockets
Sep 26 02:07:53 dist-test-slave-03rn krb5kdc[27308](info): commencing operation
krb5kdc: starting...
W20260926 02:07:54.169622 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:07:54 dist-test-slave-03rn krb5kdc[27308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:07:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:07:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:07:54.412992 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:42003
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:42003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:54.757812 27321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:54.758137 27321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:54.758296 27321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:54.765730 27321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:07:54.765848 27321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:54.765950 27321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:54.766039 27321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:07:54.766130 27321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:07:54.777040 27321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:42003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:42003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:54.778411 27321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:54.779778 27321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:54.792640 27327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:54.792618 27329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:54.792939 27326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:54.793385 27321 server_base.cc:1034] running on GCE node
I20260926 02:07:54.794327 27321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:54.795969 27321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:54.797248 27321 hybrid_clock.cc:648] HybridClock initialized: now 1790388474797175 us; error 62 us; skew 500 ppm
I20260926 02:07:54.797650 27321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:54 dist-test-slave-03rn krb5kdc[27308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388474, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:54.800936 27321 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:07:54.802531 27321 webserver.cc:460] Webserver started at http://127.18.159.190:39063/ using document root <none> and password file <none>
I20260926 02:07:54.803267 27321 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:54.803400 27321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:54.803726 27321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:54.807050 27321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "f7abdda0d22e4648895ea0b764810dfa"
format_stamp: "Formatted at 2026-09-26 02:07:54 on dist-test-slave-03rn"
server_key: "c8d9faf78ba03b77559477b99e6e5713"
server_key_iv: "a6cb709924c105e4f78f4f6af0807729"
server_key_version: "encryptionkey@0"
I20260926 02:07:54.807857 27321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "f7abdda0d22e4648895ea0b764810dfa"
format_stamp: "Formatted at 2026-09-26 02:07:54 on dist-test-slave-03rn"
server_key: "c8d9faf78ba03b77559477b99e6e5713"
server_key_iv: "a6cb709924c105e4f78f4f6af0807729"
server_key_version: "encryptionkey@0"
I20260926 02:07:54.813536 27321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:07:54.817294 27336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:54.818653 27321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:07:54.818862 27321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "f7abdda0d22e4648895ea0b764810dfa"
format_stamp: "Formatted at 2026-09-26 02:07:54 on dist-test-slave-03rn"
server_key: "c8d9faf78ba03b77559477b99e6e5713"
server_key_iv: "a6cb709924c105e4f78f4f6af0807729"
server_key_version: "encryptionkey@0"
I20260926 02:07:54.819052 27321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:54.857093 27321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:54.861763 27321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:54.862128 27321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:54.882212 27321 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:42003
I20260926 02:07:54.882230 27387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:42003 every 8 connection(s)
I20260926 02:07:54.883685 27321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:07:54.886520 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 27321
I20260926 02:07:54.886794 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:07:54.887455 19070 external_mini_cluster.cc:1428] Setting key e2f3d0dda18a115d7fbe5d93b4447d39
I20260926 02:07:54.888002 27388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:07:54.904033 27388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa: Bootstrap starting.
Sep 26 02:07:54 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:54.907521 27388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa: Neither blocks nor log segments found. Creating new log.
I20260926 02:07:54.908797 27388 log.cc:824] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:54.912600 27388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa: No bootstrap required, opened a new log
I20260926 02:07:54.913697 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:54.889705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60184 (local address 127.18.159.190:42003)
0926 02:07:54.890567 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:54.890579 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:54.890620 (+ 41us) server_negotiation.cc:407] Connection header received
0926 02:07:54.893150 (+ 2530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:54.893177 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:54.893649 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:54.894030 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:54.896094 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:54.897542 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:54.899434 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:54.900273 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:54.900439 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:54.905419 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:54.905428 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:54.905444 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:54.905496 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:54.908837 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:54.909888 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:54.909897 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:54.909906 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:54.910170 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:54.910709 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:54.910714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:54.910716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:54.911303 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:54.911546 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:54.912041 (+ 495us) server_negotiation.cc:299] Negotiation successful
0926 02:07:54.912562 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":409,"threads_started":1}
I20260926 02:07:54.920647 27388 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7abdda0d22e4648895ea0b764810dfa" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 42003 } }
I20260926 02:07:54.921077 27388 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:07:54.921193 27388 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7abdda0d22e4648895ea0b764810dfa, State: Initialized, Role: FOLLOWER
I20260926 02:07:54.921933 27388 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7abdda0d22e4648895ea0b764810dfa" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 42003 } }
I20260926 02:07:54.922260 27388 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:07:54.922410 27388 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:07:54.922598 27388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:07:54.925127 27388 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7abdda0d22e4648895ea0b764810dfa" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 42003 } }
I20260926 02:07:54.925685 27388 leader_election.cc:302] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7abdda0d22e4648895ea0b764810dfa; no voters:
I20260926 02:07:54.926381 27388 leader_election.cc:288] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:07:54.926540 27393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:07:54.927340 27393 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [term 1 LEADER]: Becoming Leader. State: Replica: f7abdda0d22e4648895ea0b764810dfa, State: Running, Role: LEADER
I20260926 02:07:54.927904 27393 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7abdda0d22e4648895ea0b764810dfa" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 42003 } }
I20260926 02:07:54.928593 27388 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:07:54.931071 27394 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7abdda0d22e4648895ea0b764810dfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7abdda0d22e4648895ea0b764810dfa" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 42003 } } }
I20260926 02:07:54.931182 27395 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7abdda0d22e4648895ea0b764810dfa. Latest consensus state: current_term: 1 leader_uuid: "f7abdda0d22e4648895ea0b764810dfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7abdda0d22e4648895ea0b764810dfa" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 42003 } } }
I20260926 02:07:54.931723 27394 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:54.931783 27395 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:54.933885 27400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:07:54.940176 27400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:07:54.962889 27400 catalog_manager.cc:1347] Generated new cluster ID: 094c039b31de45e39d316e427b75cfab
I20260926 02:07:54.963059 27400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:07:54.978519 27400 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:07:54.980616 27400 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:07:54.995105 27400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa: Generated new TSK 0
I20260926 02:07:54.996357 27400 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.159.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
I20260926 02:07:55.057919 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:42003
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:55.412021 27414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:55.412343 27414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:55.412544 27414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:55.420373 27414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:55.420487 27414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:55.420605 27414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:07:55.431008 27414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:42003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:55.432413 27414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:55.433897 27414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:55.447456 27422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:55.447549 27419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:55.447522 27420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:55.448786 27414 server_base.cc:1034] running on GCE node
I20260926 02:07:55.449478 27414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:55.450541 27414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:55.451829 27414 hybrid_clock.cc:648] HybridClock initialized: now 1790388475451764 us; error 57 us; skew 500 ppm
I20260926 02:07:55.452265 27414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:55 dist-test-slave-03rn krb5kdc[27308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388475, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:55.455592 27414 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:07:55.457283 27414 webserver.cc:460] Webserver started at http://127.18.159.129:40683/ using document root <none> and password file <none>
I20260926 02:07:55.458042 27414 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:55.458196 27414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:55.458518 27414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:55.461655 27414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "b9e91a1cf9724c3cbaa327af3cfe71ec"
format_stamp: "Formatted at 2026-09-26 02:07:55 on dist-test-slave-03rn"
server_key: "83801d976fdfecd8ba9c39b036c80a53"
server_key_iv: "2939140b3833b63257748a4f40edee2f"
server_key_version: "encryptionkey@0"
I20260926 02:07:55.462507 27414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "b9e91a1cf9724c3cbaa327af3cfe71ec"
format_stamp: "Formatted at 2026-09-26 02:07:55 on dist-test-slave-03rn"
server_key: "83801d976fdfecd8ba9c39b036c80a53"
server_key_iv: "2939140b3833b63257748a4f40edee2f"
server_key_version: "encryptionkey@0"
I20260926 02:07:55.468102 27414 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:07:55.472275 27429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:55.473899 27414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:07:55.474135 27414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "b9e91a1cf9724c3cbaa327af3cfe71ec"
format_stamp: "Formatted at 2026-09-26 02:07:55 on dist-test-slave-03rn"
server_key: "83801d976fdfecd8ba9c39b036c80a53"
server_key_iv: "2939140b3833b63257748a4f40edee2f"
server_key_version: "encryptionkey@0"
I20260926 02:07:55.474409 27414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:55.501832 27414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:55.506021 27414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:55.506333 27414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:55.507334 27414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:55.508785 27414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:55.508895 27414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:55.509044 27414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:55.509115 27414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:55.545228 27414 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:33563
I20260926 02:07:55.545326 27541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:33563 every 8 connection(s)
I20260926 02:07:55.546614 27414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:07:55.551914 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 27414
I20260926 02:07:55.552132 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:07:55.553026 19070 external_mini_cluster.cc:1428] Setting key a9aa37bd45f5c6f290b6139a1ce22079
Sep 26 02:07:55 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388475, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:55.572985 27544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:55.549758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:33775 (local address 127.18.159.190:42003)
0926 02:07:55.550242 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:55.550246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:55.551970 (+ 1724us) server_negotiation.cc:407] Connection header received
0926 02:07:55.554660 (+ 2690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:55.554665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:55.554769 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:55.554872 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:55.557250 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:55.557993 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:55.560758 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:55.561199 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:55.561314 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:55.565501 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:55.565512 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:55.565520 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:55.565572 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:55.569229 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:55.570110 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:55.570119 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:55.570125 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:55.570283 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:55.570764 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:55.570773 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:55.570780 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:55.571163 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:55.571312 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:55.572129 (+ 817us) server_negotiation.cc:299] Negotiation successful
0926 02:07:55.572459 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":255,"threads_started":1}
I20260926 02:07:55.573763 27545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:55.550752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.129:33775)
0926 02:07:55.551712 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:55.551780 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:55.554254 (+ 2474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:55.555237 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:55.555252 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:55.556065 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:55.557030 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:55.557061 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:55.558184 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:55.558192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:55.560525 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:55.560545 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:55.561487 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:55.561496 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:55.561849 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:55.562708 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:55.562745 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:55.565190 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:55.569439 (+ 4249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:55.569448 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:55.569464 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:55.569943 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:55.570413 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:55.570416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:55.570418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:55.570622 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:55.571299 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:55.571306 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:55.571680 (+ 374us) client_negotiation.cc:769] Sending connection context
0926 02:07:55.571956 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 02:07:55.572526 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":336,"threads_started":1}
I20260926 02:07:55.575760 27542 heartbeater.cc:347] Connected to a master server at 127.18.159.190:42003
I20260926 02:07:55.576089 27542 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:55.576869 27542 heartbeater.cc:511] Master 127.18.159.190:42003 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.159.130@KRBTEST.COM; defaulting to no policy
I20260926 02:07:55.579487 27353 ts_manager.cc:194] Registered new tserver with Master: b9e91a1cf9724c3cbaa327af3cfe71ec (127.18.159.129:33563)
I20260926 02:07:55.581111 27353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:33775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
I20260926 02:07:55.613287 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:42003
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:55.981084 27548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:55.981441 27548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:55.981616 27548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:55.990316 27548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:55.990487 27548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:55.990720 27548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:07:56.002753 27548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:42003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:56.004103 27548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:56.005530 27548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:56.019717 27553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:56.019667 27554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:56.019616 27556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:56.020833 27548 server_base.cc:1034] running on GCE node
I20260926 02:07:56.021610 27548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:56.022558 27548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:56.023862 27548 hybrid_clock.cc:648] HybridClock initialized: now 1790388476023820 us; error 47 us; skew 500 ppm
I20260926 02:07:56.024277 27548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:56 dist-test-slave-03rn krb5kdc[27308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388476, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:56.036450 27548 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:07:56.038199 27548 webserver.cc:460] Webserver started at http://127.18.159.130:43915/ using document root <none> and password file <none>
I20260926 02:07:56.038988 27548 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:56.039124 27548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:56.039469 27548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:56.042716 27548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "e29719448b03402791db3bf5235754d6"
format_stamp: "Formatted at 2026-09-26 02:07:56 on dist-test-slave-03rn"
server_key: "fd0bf139b77d9547210e959696cf83e3"
server_key_iv: "ff7c0e0547a32edeee4938dc6685fd8f"
server_key_version: "encryptionkey@0"
I20260926 02:07:56.043628 27548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "e29719448b03402791db3bf5235754d6"
format_stamp: "Formatted at 2026-09-26 02:07:56 on dist-test-slave-03rn"
server_key: "fd0bf139b77d9547210e959696cf83e3"
server_key_iv: "ff7c0e0547a32edeee4938dc6685fd8f"
server_key_version: "encryptionkey@0"
I20260926 02:07:56.049614 27548 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 02:07:56.053783 27563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:56.055370 27548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260926 02:07:56.055611 27548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "e29719448b03402791db3bf5235754d6"
format_stamp: "Formatted at 2026-09-26 02:07:56 on dist-test-slave-03rn"
server_key: "fd0bf139b77d9547210e959696cf83e3"
server_key_iv: "ff7c0e0547a32edeee4938dc6685fd8f"
server_key_version: "encryptionkey@0"
I20260926 02:07:56.055835 27548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:56.084260 27548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:56.089133 27548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:56.089457 27548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:56.090538 27548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:56.091985 27548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:56.092062 27548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:56.092226 27548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:56.092278 27548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 02:07:56.127341 27548 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:36229
I20260926 02:07:56.127447 27675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:36229 every 8 connection(s)
I20260926 02:07:56.128952 27548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:07:56.139115 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 27548
I20260926 02:07:56.139424 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:07:56.139938 19070 external_mini_cluster.cc:1428] Setting key d721db139d57bf6d0b24bfbcbce5a9c9
Sep 26 02:07:56 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388476, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:56.151904 27678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:56.131813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:42291 (local address 127.18.159.190:42003)
0926 02:07:56.132295 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:56.132300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:56.133597 (+ 1297us) server_negotiation.cc:407] Connection header received
0926 02:07:56.135770 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:56.135777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:56.135882 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:56.136025 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:56.137846 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:56.138570 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:56.141140 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:56.141682 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:56.141836 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:56.145764 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:56.145770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:56.145774 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:56.145810 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:56.148619 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:56.149442 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:56.149450 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:56.149458 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:56.149585 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:56.150072 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:56.150079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:56.150087 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:56.150365 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:56.150520 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:56.151294 (+ 774us) server_negotiation.cc:299] Negotiation successful
0926 02:07:56.151534 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":202,"threads_started":1}
I20260926 02:07:56.152967 27679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:56.132548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.130:42291)
0926 02:07:56.133376 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:56.133432 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:56.135390 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:56.136353 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:56.136365 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:56.136989 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:56.137624 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:56.137642 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:56.138791 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:56.138795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:56.140973 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:56.140996 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:56.141942 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:56.141950 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:56.142221 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:56.143006 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:56.143031 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:56.145459 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:56.148797 (+ 3338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:56.148807 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:56.148836 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:56.149297 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:56.149715 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:56.149721 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:56.149724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:56.149948 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:56.150501 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:56.150509 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:56.150895 (+ 386us) client_negotiation.cc:769] Sending connection context
0926 02:07:56.151176 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 02:07:56.151763 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":336,"threads_started":1}
I20260926 02:07:56.154587 27676 heartbeater.cc:347] Connected to a master server at 127.18.159.190:42003
I20260926 02:07:56.154958 27676 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:56.155884 27676 heartbeater.cc:511] Master 127.18.159.190:42003 requested a full tablet report, sending...
I20260926 02:07:56.157866 27353 ts_manager.cc:194] Registered new tserver with Master: e29719448b03402791db3bf5235754d6 (127.18.159.130:36229)
I20260926 02:07:56.159199 27353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:42291
WARNING: no policy specified for kudu/127.18.159.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
I20260926 02:07:56.193353 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:42003
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:56.545722 27682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:56.546001 27682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:56.546161 27682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:56.553928 27682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:56.554046 27682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:56.554231 27682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:07:56.564815 27682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:42003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:56.566145 27682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:56.567480 27682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:56.581930 27687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:56.581836 27688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:56.582185 27690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:56.583137 27682 server_base.cc:1034] running on GCE node
I20260926 02:07:56.583827 27682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:56.584978 27682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:56.585539 27542 heartbeater.cc:503] Master 127.18.159.190:42003 was elected leader, sending a full tablet report...
I20260926 02:07:56.586222 27682 hybrid_clock.cc:648] HybridClock initialized: now 1790388476586184 us; error 53 us; skew 500 ppm
I20260926 02:07:56.586639 27682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:56 dist-test-slave-03rn krb5kdc[27308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388476, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:56.590189 27682 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:07:56.591760 27682 webserver.cc:460] Webserver started at http://127.18.159.131:35131/ using document root <none> and password file <none>
I20260926 02:07:56.592561 27682 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:56.592669 27682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:56.592940 27682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:07:56.596004 27682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "e3058892adc44fd2859b0060886632fc"
format_stamp: "Formatted at 2026-09-26 02:07:56 on dist-test-slave-03rn"
server_key: "66ce644396e5354d9359deccbdf1700a"
server_key_iv: "1dd7dac87f35dee62697074a8c135a61"
server_key_version: "encryptionkey@0"
I20260926 02:07:56.596863 27682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "e3058892adc44fd2859b0060886632fc"
format_stamp: "Formatted at 2026-09-26 02:07:56 on dist-test-slave-03rn"
server_key: "66ce644396e5354d9359deccbdf1700a"
server_key_iv: "1dd7dac87f35dee62697074a8c135a61"
server_key_version: "encryptionkey@0"
I20260926 02:07:56.602831 27682 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:07:56.606637 27697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:56.608142 27682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:07:56.608433 27682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "e3058892adc44fd2859b0060886632fc"
format_stamp: "Formatted at 2026-09-26 02:07:56 on dist-test-slave-03rn"
server_key: "66ce644396e5354d9359deccbdf1700a"
server_key_iv: "1dd7dac87f35dee62697074a8c135a61"
server_key_version: "encryptionkey@0"
I20260926 02:07:56.608644 27682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:56.626490 27682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:56.631438 27682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:56.631752 27682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:56.632896 27682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:56.634377 27682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:56.634455 27682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:56.634588 27682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:56.634639 27682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:56.670063 27682 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:45373
I20260926 02:07:56.670105 27809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:45373 every 8 connection(s)
I20260926 02:07:56.671489 27682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:07:56.677279 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 27682
I20260926 02:07:56.677466 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:07:56.678032 19070 external_mini_cluster.cc:1428] Setting key 4ce44e69bccf1f67b973f4e697db5a20
Sep 26 02:07:56 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388476, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:07:56.695470 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:56.674313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:36353 (local address 127.18.159.190:42003)
0926 02:07:56.674808 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:56.674814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:56.675978 (+ 1164us) server_negotiation.cc:407] Connection header received
0926 02:07:56.678688 (+ 2710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:56.678695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:56.678800 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:56.678937 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:56.681100 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:56.682037 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:56.683754 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:56.684241 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:56.684382 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:56.688669 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:56.688675 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:56.688679 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:56.688716 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:56.691405 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:56.692457 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:56.692461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:56.692464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:56.692654 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:56.693327 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:56.693332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:56.693334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:56.693753 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:56.693874 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:56.694965 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0926 02:07:56.695171 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":228,"threads_started":1}
I20260926 02:07:56.696601 27813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:56.675007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.131:36353)
0926 02:07:56.675790 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:56.675839 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:56.678180 (+ 2341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:56.679248 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:56.679257 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:56.680109 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:56.680913 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:56.680932 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:56.682208 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:56.682212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:56.683537 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:56.683551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:56.684508 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:56.684514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:56.684780 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:56.685568 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:56.685593 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:56.688377 (+ 2784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:07:56.691589 (+ 3212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:56.691598 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:56.691614 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:56.692249 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:56.692785 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:07:56.692789 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:07:56.692825 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:07:56.693174 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:07:56.693931 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:07:56.693938 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:07:56.694445 (+ 507us) client_negotiation.cc:769] Sending connection context
0926 02:07:56.694804 (+ 359us) client_negotiation.cc:240] Negotiation successful
0926 02:07:56.695256 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":323,"threads_started":1}
I20260926 02:07:56.698274 27810 heartbeater.cc:347] Connected to a master server at 127.18.159.190:42003
I20260926 02:07:56.698609 27810 heartbeater.cc:464] Registering TS with master...
I20260926 02:07:56.699450 27810 heartbeater.cc:511] Master 127.18.159.190:42003 requested a full tablet report, sending...
I20260926 02:07:56.701072 27353 ts_manager.cc:194] Registered new tserver with Master: e3058892adc44fd2859b0060886632fc (127.18.159.131:45373)
I20260926 02:07:56.701956 27353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:36353
I20260926 02:07:56.707396 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.159.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.159.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.159.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.129.keytab.
Entry for principal oryx/127.18.159.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.129.keytab.
WARNING: no policy specified for oryx/127.18.159.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.159.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.159.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.130.keytab.
Entry for principal oryx/127.18.159.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.130.keytab.
WARNING: no policy specified for oryx/127.18.159.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.159.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.159.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.131.keytab.
Entry for principal oryx/127.18.159.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.131.keytab.
I20260926 02:07:56.855710 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 27414
Sep 26 02:07:56 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:07:56.873689 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:56.857094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42362 (local address 127.18.159.129:33563)
0926 02:07:56.857783 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:56.857790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:56.857818 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:07:56.857879 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:56.857886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:56.858110 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:56.858345 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:56.859837 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:56.861076 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:56.862364 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:56.862943 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:56.863094 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:56.867370 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:56.867384 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:56.867405 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:56.867449 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:56.870462 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:56.871293 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:56.871301 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:56.871303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:56.871450 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:56.871838 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:56.871843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:56.871848 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:56.872360 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:56.872690 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:56.873038 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 02:07:56.873337 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":362,"threads_started":1}
I20260926 02:07:56.875018 27537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:56.960212 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 27414
I20260926 02:07:56.976934 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 27548
Sep 26 02:07:56 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.130@KRBTEST.COM
I20260926 02:07:56.994194 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:56.977889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43628 (local address 127.18.159.130:36229)
0926 02:07:56.978472 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:56.978480 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:56.978506 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:07:56.978566 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:56.978572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:56.978798 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:56.978932 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:56.980294 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:56.981356 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:56.982651 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:56.983252 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:56.983366 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:56.987904 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:56.987918 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:56.987935 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:56.987977 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:56.991042 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:56.991884 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:56.991893 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:56.991895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:56.992044 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:56.992500 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:56.992506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:56.992510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:56.992883 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:56.993225 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:56.993533 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 02:07:56.993809 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":306,"threads_started":1}
I20260926 02:07:56.995733 27671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:57.078141 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 27548
I20260926 02:07:57.093967 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 27682
Sep 26 02:07:57 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.131@KRBTEST.COM
I20260926 02:07:57.111864 27825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:57.095092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55608 (local address 127.18.159.131:45373)
0926 02:07:57.095711 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:57.095718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:57.095748 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:07:57.095819 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:57.095826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:57.096040 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:57.096260 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:57.097693 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:57.098755 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:57.100099 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:57.100758 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:57.100922 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:57.105631 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:57.105646 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:57.105663 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:57.105703 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:57.108814 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:57.109632 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:57.109641 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:57.109643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:57.109782 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:57.110214 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:57.110219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:57.110223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:57.110587 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:57.110902 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:57.111227 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 02:07:57.111491 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":328,"threads_started":1}
I20260926 02:07:57.113493 27805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:57.199399 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 27682
I20260926 02:07:57.215318 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 27321
I20260926 02:07:57.216741 27383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:07:57.321545 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 27321
I20260926 02:07:57.342489 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:42003
--webserver_interface=127.18.159.190
--webserver_port=39063
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:42003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:07:57.688329 27828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:57.688651 27828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:57.688812 27828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:57.696761 27828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:07:57.696945 27828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:57.697058 27828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:57.697165 27828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:07:57.697242 27828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:07:57.707737 27828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:42003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:42003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=39063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:57.709287 27828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:57.710668 27828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:57.723271 27834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:57.723528 27833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:57.723317 27836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:57.725831 27828 server_base.cc:1034] running on GCE node
I20260926 02:07:57.726912 27828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:57.728790 27828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:57.730054 27828 hybrid_clock.cc:648] HybridClock initialized: now 1790388477730005 us; error 62 us; skew 500 ppm
I20260926 02:07:57.730458 27828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:57 dist-test-slave-03rn krb5kdc[27308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388477, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:57.734026 27828 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:07:57.735857 27828 webserver.cc:460] Webserver started at http://127.18.159.190:39063/ using document root <none> and password file <none>
I20260926 02:07:57.736770 27828 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:57.736951 27828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:57.742830 27828 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:07:57.746124 27843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:57.747712 27828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:07:57.747933 27828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "f7abdda0d22e4648895ea0b764810dfa"
format_stamp: "Formatted at 2026-09-26 02:07:54 on dist-test-slave-03rn"
server_key: "c8d9faf78ba03b77559477b99e6e5713"
server_key_iv: "a6cb709924c105e4f78f4f6af0807729"
server_key_version: "encryptionkey@0"
I20260926 02:07:57.749049 27828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:57.766523 27828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:57.771107 27828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:57.771399 27828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:57.791352 27828 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:42003
I20260926 02:07:57.791368 27894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:42003 every 8 connection(s)
I20260926 02:07:57.793063 27828 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:07:57.796295 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 27828
I20260926 02:07:57.797259 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:33563
--local_ip_for_outbound_sockets=127.18.159.129
--tserver_master_addrs=127.18.159.190:42003
--webserver_port=40683
--webserver_interface=127.18.159.129
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.159.129
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:07:57.802361 27895 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:07:57.806857 27895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa: Bootstrap starting.
I20260926 02:07:57.826871 27895 log.cc:824] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa: Log is configured to *not* fsync() on all Append() calls
I20260926 02:07:57.835666 27895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:07:57.836299 27895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa: Bootstrap complete.
I20260926 02:07:57.845570 27895 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7abdda0d22e4648895ea0b764810dfa" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 42003 } }
I20260926 02:07:57.846398 27895 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7abdda0d22e4648895ea0b764810dfa, State: Initialized, Role: FOLLOWER
I20260926 02:07:57.847097 27895 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7abdda0d22e4648895ea0b764810dfa" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 42003 } }
I20260926 02:07:57.847337 27895 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:07:57.847450 27895 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:07:57.847642 27895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:07:57.851930 27895 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7abdda0d22e4648895ea0b764810dfa" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 42003 } }
I20260926 02:07:57.852638 27895 leader_election.cc:302] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7abdda0d22e4648895ea0b764810dfa; no voters:
I20260926 02:07:57.853366 27895 leader_election.cc:288] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 02:07:57.853505 27899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [term 2 FOLLOWER]: Leader election won for term 2
I20260926 02:07:57.856505 27895 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:07:57.856557 27899 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [term 2 LEADER]: Becoming Leader. State: Replica: f7abdda0d22e4648895ea0b764810dfa, State: Running, Role: LEADER
I20260926 02:07:57.857229 27899 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7abdda0d22e4648895ea0b764810dfa" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 42003 } }
I20260926 02:07:57.860131 27900 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f7abdda0d22e4648895ea0b764810dfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7abdda0d22e4648895ea0b764810dfa" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 42003 } } }
I20260926 02:07:57.860745 27901 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7abdda0d22e4648895ea0b764810dfa. Latest consensus state: current_term: 2 leader_uuid: "f7abdda0d22e4648895ea0b764810dfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7abdda0d22e4648895ea0b764810dfa" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 42003 } } }
I20260926 02:07:57.861136 27900 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:57.861160 27901 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa [sys.catalog]: This master's current role is: LEADER
I20260926 02:07:57.872843 27905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:07:57.879745 27905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:07:57.881457 27905 catalog_manager.cc:1259] Loaded cluster ID: 094c039b31de45e39d316e427b75cfab
I20260926 02:07:57.881590 27905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:07:57.885967 27905 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:07:57.886487 27917 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 02:07:57.887967 27905 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f7abdda0d22e4648895ea0b764810dfa: Loaded TSK: 0
I20260926 02:07:57.889736 27905 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 02:07:58.166978 27897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:58.167281 27897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:58.167443 27897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:58.175844 27897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:58.175981 27897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:58.176224 27897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:07:58.186614 27897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.129.keytab
--principal=oryx/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:33563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=40683
--tserver_master_addrs=127.18.159.190:42003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:58.187997 27897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:58.189368 27897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:58.203246 27923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:58.203572 27925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:58.204023 27922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:58.204414 27897 server_base.cc:1034] running on GCE node
I20260926 02:07:58.205091 27897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:58.206173 27897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:58.207412 27897 hybrid_clock.cc:648] HybridClock initialized: now 1790388478207343 us; error 62 us; skew 500 ppm
I20260926 02:07:58.207742 27897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:58 dist-test-slave-03rn krb5kdc[27308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388478, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:58.211086 27897 init.cc:377] Logged in from keytab as oryx/127.18.159.129@KRBTEST.COM (short username oryx)
I20260926 02:07:58.212997 27897 webserver.cc:460] Webserver started at http://127.18.159.129:40683/ using document root <none> and password file <none>
I20260926 02:07:58.213771 27897 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:58.213928 27897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:58.220569 27897 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:07:58.223992 27932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:58.225695 27897 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:07:58.225914 27897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "b9e91a1cf9724c3cbaa327af3cfe71ec"
format_stamp: "Formatted at 2026-09-26 02:07:55 on dist-test-slave-03rn"
server_key: "83801d976fdfecd8ba9c39b036c80a53"
server_key_iv: "2939140b3833b63257748a4f40edee2f"
server_key_version: "encryptionkey@0"
I20260926 02:07:58.226950 27897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:58.247145 27897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:58.251999 27897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:58.252398 27897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:58.253600 27897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:58.255239 27897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:58.255409 27897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:58.255573 27897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:58.255666 27897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:58.291846 27897 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:33563
I20260926 02:07:58.291945 28044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:33563 every 8 connection(s)
I20260926 02:07:58.293502 27897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:07:58.293901 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 27897
I20260926 02:07:58.294574 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:36229
--local_ip_for_outbound_sockets=127.18.159.130
--tserver_master_addrs=127.18.159.190:42003
--webserver_port=43915
--webserver_interface=127.18.159.130
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.159.130
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:07:58 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:07:58 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:07:58.317158 28048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:58.297724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.129:39221)
0926 02:07:58.300779 (+ 3055us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:58.300836 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:58.302939 (+ 2103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:58.304955 (+ 2016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:58.304964 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:58.305523 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:58.306106 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:58.306123 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.308602 (+ 2479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.308606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:58.309888 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:58.309900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.311424 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.311429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:58.311635 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:58.312422 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:58.312444 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:58.316093 (+ 3649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2753,"thread_start_us":2549,"threads_started":1}
W20260926 02:07:58.317890 28045 heartbeater.cc:644] Failed to heartbeat to 127.18.159.190:42003 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.159.190:42003: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
W20260926 02:07:58.319000 28047 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:58.297420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:39221 (local address 127.18.159.190:42003)
0926 02:07:58.298150 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:58.298163 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:58.301007 (+ 2844us) server_negotiation.cc:407] Connection header received
0926 02:07:58.303744 (+ 2737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:58.303769 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:58.304342 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:58.304780 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:58.306343 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.308422 (+ 2079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.310058 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.311194 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.311421 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:58.316659 (+ 5238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.129:39221: BlockingRecv error: recv got EOF from 127.18.159.129:39221 (error 108)
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":287,"threads_started":1}
Sep 26 02:07:58 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:07:58 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:07:58.327045 28048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:58.318534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.129:37779)
0926 02:07:58.318824 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:58.318845 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:58.318967 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:58.319643 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:58.319647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:58.320125 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:58.320342 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:58.320352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.321368 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.321374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:58.322279 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:58.322290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.322994 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.323000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:58.323177 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:58.323819 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:58.323839 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:58.326746 (+ 2907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172}
W20260926 02:07:58.327224 28047 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:58.318807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:37779 (local address 127.18.159.190:42003)
0926 02:07:58.319374 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:58.319378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:58.319393 (+ 15us) server_negotiation.cc:407] Connection header received
0926 02:07:58.319441 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:58.319445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:58.319499 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:58.319612 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:58.320486 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.321215 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.322432 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.322847 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.322983 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:58.326973 (+ 3990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.129:37779: BlockingRecv error: recv got EOF from 127.18.159.129:37779 (error 108)
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":1092,"threads_started":1}
Sep 26 02:07:58 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:07:58 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:07:58.337100 28048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:58.327823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.129:57559)
0926 02:07:58.328028 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:58.328048 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:58.328197 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:58.328530 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:58.328535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:58.328909 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:58.329097 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:58.329108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.330111 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.330117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:58.331565 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:58.331582 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.332432 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.332438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:58.332631 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:58.333256 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:58.333277 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:58.336760 (+ 3483us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 02:07:58.337282 28047 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:58.328060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:57559 (local address 127.18.159.190:42003)
0926 02:07:58.328263 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:58.328267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:58.328283 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:07:58.328335 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:58.328339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:58.328392 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:58.328469 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:58.329246 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.329963 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.331760 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.332280 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.332389 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:58.337004 (+ 4615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.129:57559: BlockingRecv error: recv got EOF from 127.18.159.129:57559 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 02:07:58.337466 28045 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:07:58.655113 28049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:58.655428 28049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:58.655599 28049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:58.663936 28049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:58.664145 28049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:58.664373 28049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:07:58.675259 28049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.130.keytab
--principal=oryx/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:36229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=43915
--tserver_master_addrs=127.18.159.190:42003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:58.676738 28049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:58.678186 28049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:58.693754 28055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:58.693730 28058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:58.693754 28056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:58.694751 28049 server_base.cc:1034] running on GCE node
I20260926 02:07:58.695576 28049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:58.696864 28049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:58.698148 28049 hybrid_clock.cc:648] HybridClock initialized: now 1790388478698071 us; error 74 us; skew 500 ppm
I20260926 02:07:58.698534 28049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:58 dist-test-slave-03rn krb5kdc[27308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388478, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:58.702400 28049 init.cc:377] Logged in from keytab as oryx/127.18.159.130@KRBTEST.COM (short username oryx)
I20260926 02:07:58.704340 28049 webserver.cc:460] Webserver started at http://127.18.159.130:43915/ using document root <none> and password file <none>
I20260926 02:07:58.705161 28049 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:58.705353 28049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:58.711473 28049 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 02:07:58.714691 28065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:58.716413 28049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:07:58.716626 28049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "e29719448b03402791db3bf5235754d6"
format_stamp: "Formatted at 2026-09-26 02:07:56 on dist-test-slave-03rn"
server_key: "fd0bf139b77d9547210e959696cf83e3"
server_key_iv: "ff7c0e0547a32edeee4938dc6685fd8f"
server_key_version: "encryptionkey@0"
I20260926 02:07:58.717731 28049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:58.743409 28049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:58.747807 28049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:58.748122 28049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:58.749176 28049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:58.750620 28049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:58.750754 28049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:58.751026 28049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:58.751132 28049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:58.787086 28049 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:36229
I20260926 02:07:58.787145 28177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:36229 every 8 connection(s)
I20260926 02:07:58.788695 28049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:07:58.793632 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 28049
I20260926 02:07:58.794341 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:45373
--local_ip_for_outbound_sockets=127.18.159.131
--tserver_master_addrs=127.18.159.190:42003
--webserver_port=35131
--webserver_interface=127.18.159.131
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.159.131
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:07:58 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:07:58 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 02:07:58.807149 28047 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:58.791525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:55989 (local address 127.18.159.190:42003)
0926 02:07:58.791736 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:58.791743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:58.793131 (+ 1388us) server_negotiation.cc:407] Connection header received
0926 02:07:58.795290 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:58.795295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:58.795392 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:58.795491 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:58.797220 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.798027 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.799731 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.800262 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.800411 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:58.806890 (+ 6479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.130:55989: BlockingRecv error: recv got EOF from 127.18.159.130:55989 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:07:58.808389 28180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:58.792032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.130:55989)
0926 02:07:58.792942 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:58.792997 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:58.794980 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:58.795795 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:58.795805 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:58.796415 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:58.797020 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:58.797038 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.798256 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.798260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:58.799551 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:58.799564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.800513 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.800521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:58.800771 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:58.801521 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:58.801550 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:58.806652 (+ 5102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":427,"threads_started":1}
W20260926 02:07:58.809242 28178 heartbeater.cc:644] Failed to heartbeat to 127.18.159.190:42003 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.159.190:42003: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Sep 26 02:07:58 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:07:58 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:07:58.818632 28180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:58.809911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.130:40525)
0926 02:07:58.810088 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:58.810109 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:58.810225 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:58.810548 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:58.810551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:58.810899 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:58.811047 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:58.811055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.812630 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.812635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:58.813565 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:58.813579 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.814492 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.814498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:58.814690 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:58.815342 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:58.815361 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:58.818325 (+ 2964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 02:07:58.818836 28047 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:58.810029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:40525 (local address 127.18.159.190:42003)
0926 02:07:58.810223 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:58.810231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:58.810256 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:07:58.810322 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:58.810329 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:58.810410 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:58.810518 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:58.811211 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.812452 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.813786 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.814307 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.814458 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:58.818567 (+ 4109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.130:40525: BlockingRecv error: recv got EOF from 127.18.159.130:40525 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 02:07:58 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:07:58 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:07:58.828402 28180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:58.819746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.130:53587)
0926 02:07:58.820004 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:58.820026 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:58.820147 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:58.820576 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:58.820581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:58.820919 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:58.821070 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:58.821078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.822157 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.822162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:58.823181 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:58.823192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.823903 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.823908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:58.824086 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:58.824793 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:58.824812 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:58.828035 (+ 3223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121}
W20260926 02:07:58.828552 28047 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:58.820075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:53587 (local address 127.18.159.190:42003)
0926 02:07:58.820285 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:58.820292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:58.820329 (+ 37us) server_negotiation.cc:407] Connection header received
0926 02:07:58.820383 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:58.820387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:58.820441 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:58.820550 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:58.821228 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.822020 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.823318 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:58.823753 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:58.823892 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:58.828318 (+ 4426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.130:53587: BlockingRecv error: recv got EOF from 127.18.159.130:53587 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 02:07:58.828954 28178 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:07:59.143280 28181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:07:59.143581 28181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:07:59.143765 28181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:07:59.151302 28181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:07:59.151424 28181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:07:59.151623 28181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:07:59.161975 28181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/krb5kdc/oryx_127.18.159.131.keytab
--principal=oryx/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:45373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=35131
--tserver_master_addrs=127.18.159.190:42003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:07:59.163362 28181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:07:59.164878 28181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:07:59.179126 28187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:59.179175 28189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:07:59.179260 28186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:07:59.180074 28181 server_base.cc:1034] running on GCE node
I20260926 02:07:59.180804 28181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:07:59.181951 28181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:07:59.183223 28181 hybrid_clock.cc:648] HybridClock initialized: now 1790388479183149 us; error 65 us; skew 500 ppm
I20260926 02:07:59.183619 28181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:07:59 dist-test-slave-03rn krb5kdc[27308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388479, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:07:59.186968 28181 init.cc:377] Logged in from keytab as oryx/127.18.159.131@KRBTEST.COM (short username oryx)
I20260926 02:07:59.188547 28181 webserver.cc:460] Webserver started at http://127.18.159.131:35131/ using document root <none> and password file <none>
I20260926 02:07:59.189226 28181 fs_manager.cc:359] Metadata directory not provided
I20260926 02:07:59.189340 28181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:07:59.195370 28181 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:07:59.198416 28196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:59.200059 28181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:07:59.200316 28181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "e3058892adc44fd2859b0060886632fc"
format_stamp: "Formatted at 2026-09-26 02:07:56 on dist-test-slave-03rn"
server_key: "66ce644396e5354d9359deccbdf1700a"
server_key_iv: "1dd7dac87f35dee62697074a8c135a61"
server_key_version: "encryptionkey@0"
I20260926 02:07:59.201329 28181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:07:59.227783 28181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:07:59.232812 28181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:07:59.233047 28181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:07:59.234030 28181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:07:59.235399 28181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:07:59.235496 28181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:59.235577 28181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:07:59.235646 28181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:07:59.271386 28181 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:45373
I20260926 02:07:59.271521 28308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:45373 every 8 connection(s)
I20260926 02:07:59.273327 28181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:07:59.273633 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 28181
Sep 26 02:07:59 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:07:59 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:07:59.294152 28047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:59.274793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60190 (local address 127.18.159.190:42003)
0926 02:07:59.274988 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:59.274994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:59.275631 (+ 637us) server_negotiation.cc:407] Connection header received
0926 02:07:59.275921 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:59.275927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:59.276019 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:59.276097 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:59.278498 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.279624 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.281166 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.281704 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.281808 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:59.285122 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:07:59.285131 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:07:59.285150 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:07:59.285217 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:07:59.289299 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:59.290362 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:59.290477 (+ 115us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:59.290497 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:59.290787 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:07:59.291370 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:07:59.291377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:07:59.291379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:07:59.292068 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:07:59.292447 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0926 02:07:59.293332 (+ 885us) server_negotiation.cc:299] Negotiation successful
0926 02:07:59.293718 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 02:07:59.294209 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:59.276836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:35293 (local address 127.18.159.190:42003)
0926 02:07:59.277347 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:59.277356 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:59.278549 (+ 1193us) server_negotiation.cc:407] Connection header received
0926 02:07:59.280966 (+ 2417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:59.280972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:59.281060 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:59.281171 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:59.283578 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.284451 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.286553 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.287442 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.287643 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:59.293958 (+ 6315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.131:35293: BlockingRecv error: recv got EOF from 127.18.159.131:35293 (error 108)
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":278,"threads_started":1}
I20260926 02:07:59.295302 28313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:59.277467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.131:35293)
0926 02:07:59.278308 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:59.278378 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:59.280660 (+ 2282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:59.281477 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:59.281487 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:59.282392 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:59.283332 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:59.283364 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.284625 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.284633 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:59.286363 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:59.286379 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.287745 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.287751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:59.288011 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:59.288759 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:59.288784 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:59.293728 (+ 4944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":341,"threads_started":1}
W20260926 02:07:59.296216 28309 heartbeater.cc:644] Failed to heartbeat to 127.18.159.190:42003 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.159.190:42003: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Sep 26 02:07:59 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:07:59 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:07:59.305951 28313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:59.296980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.131:40903)
0926 02:07:59.297177 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:59.297199 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:59.297300 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:59.298009 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:59.298014 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:59.298494 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:59.298674 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:59.298683 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.299953 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.299959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:59.300982 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:59.300995 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.301767 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.301773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:59.301971 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:59.302622 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:59.302640 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:59.305727 (+ 3087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 02:07:59.306291 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:59.297280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:40903 (local address 127.18.159.190:42003)
0926 02:07:59.297656 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:59.297663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:59.297682 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:07:59.297735 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:59.297741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:59.297818 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:59.297921 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:59.298831 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.299786 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.301148 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.301593 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.301746 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:59.305976 (+ 4230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.131:40903: BlockingRecv error: recv got EOF from 127.18.159.131:40903 (error 108)
Metrics: {"server-negotiator.queue_time_us":251}
Sep 26 02:07:59 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:07:59 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:07:59.315014 28313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:59.306696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.131:57129)
0926 02:07:59.306852 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:59.306868 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:59.306995 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:59.307362 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:59.307366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:59.307698 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:59.307850 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:59.307859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.308935 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.308942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:59.310037 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:59.310050 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.310940 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.310945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:59.311124 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:59.311815 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:59.311832 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:59.314682 (+ 2850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 02:07:59.315186 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:59.306787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:57129 (local address 127.18.159.190:42003)
0926 02:07:59.307071 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:59.307077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:59.307096 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:07:59.307155 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:59.307158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:59.307226 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:59.307335 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:59.308000 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.308782 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.310186 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.310782 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.310935 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:59.314886 (+ 3951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.131:57129: BlockingRecv error: recv got EOF from 127.18.159.131:57129 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
W20260926 02:07:59.315408 28309 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 02:07:59 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:07:59 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:07:59.347402 28314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:59.338134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.129:49563)
0926 02:07:59.338645 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:59.338675 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:59.338874 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:59.339242 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:59.339247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:59.339735 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:59.339928 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:59.339936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.341024 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.341032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:59.341991 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:59.342003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.342836 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.342843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:59.343087 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:59.343821 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:59.343839 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:59.347041 (+ 3202us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":281,"threads_started":1}
W20260926 02:07:59.347635 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:59.338198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:49563 (local address 127.18.159.190:42003)
0926 02:07:59.338434 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:59.338443 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:59.338774 (+ 331us) server_negotiation.cc:407] Connection header received
0926 02:07:59.339010 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:59.339019 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:59.339091 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:59.339207 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:59.340085 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.340871 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.342144 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.342575 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.342711 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:59.347290 (+ 4579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.129:49563: BlockingRecv error: recv got EOF from 127.18.159.129:49563 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 02:07:59 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:07:59 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:07:59.837687 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:07:59.829599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.130:58997)
0926 02:07:59.830051 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 02:07:59.830067 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:07:59.830285 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:07:59.830609 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:07:59.830612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:07:59.831031 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:07:59.831215 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:59.831224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.832271 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.832274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:59.833228 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:07:59.833239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.833922 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.833926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:07:59.834045 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:59.834624 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:07:59.834641 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:07:59.837458 (+ 2817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":240,"threads_started":1}
W20260926 02:07:59.837934 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:07:59.829634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:58997 (local address 127.18.159.190:42003)
0926 02:07:59.829823 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 02:07:59.829834 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:07:59.830182 (+ 348us) server_negotiation.cc:407] Connection header received
0926 02:07:59.830413 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:07:59.830417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:07:59.830478 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:07:59.830558 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:07:59.831377 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.832106 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.833352 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:07:59.833768 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:07:59.833866 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:07:59.837674 (+ 3808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.130:58997: BlockingRecv error: recv got EOF from 127.18.159.130:58997 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 02:08:00 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:00 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:00.325212 28316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:00.316100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.131:60299)
0926 02:08:00.316697 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:00.316718 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:00.316837 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:00.317164 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:00.317167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:00.317595 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:00.317783 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:00.317791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:00.318821 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:00.318824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:00.319805 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:00.319817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:00.320881 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:00.320890 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:00.321080 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:00.321859 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:00.321910 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:00.324986 (+ 3076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":358,"threads_started":1}
W20260926 02:08:00.325505 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:00.316283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:60299 (local address 127.18.159.190:42003)
0926 02:08:00.316468 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:00.316472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:00.316800 (+ 328us) server_negotiation.cc:407] Connection header received
0926 02:08:00.316947 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:00.316951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:00.317013 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:00.317105 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:00.317927 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:00.318687 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:00.320004 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:00.320626 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:00.320757 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:00.325226 (+ 4469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.131:60299: BlockingRecv error: recv got EOF from 127.18.159.131:60299 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 02:08:00 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:00 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:00.357106 28317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:00.348473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.129:57443)
0926 02:08:00.349035 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:00.349052 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:00.349210 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:00.349554 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:00.349561 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:00.349988 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:00.350155 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:00.350163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:00.351155 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:00.351162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:00.352059 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:00.352071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:00.352857 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:00.352863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:00.353039 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:00.353761 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:00.353780 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:00.356828 (+ 3048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":317,"threads_started":1}
W20260926 02:08:00.357362 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:00.348587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:57443 (local address 127.18.159.190:42003)
0926 02:08:00.348860 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:00.348867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:00.349139 (+ 272us) server_negotiation.cc:407] Connection header received
0926 02:08:00.349349 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:00.349353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:00.349414 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:00.349517 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:00.350301 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:00.351008 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:00.352229 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:00.352672 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:00.352811 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:00.357059 (+ 4248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.129:57443: BlockingRecv error: recv got EOF from 127.18.159.129:57443 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 26 02:08:00 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:00 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:00.847236 28318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:00.838819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.130:38981)
0926 02:08:00.839385 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:00.839401 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:00.839512 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:00.839876 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:00.839880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:00.840340 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:00.840511 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:00.840520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:00.841568 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:00.841573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:00.842518 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:00.842530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:00.843299 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:00.843304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:00.843421 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:00.844051 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:00.844069 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:00.846988 (+ 2919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":315,"threads_started":1}
W20260926 02:08:00.847434 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:00.839000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:38981 (local address 127.18.159.190:42003)
0926 02:08:00.839174 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:00.839179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:00.839484 (+ 305us) server_negotiation.cc:407] Connection header received
0926 02:08:00.839615 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:00.839619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:00.839679 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:00.839792 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:00.840670 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:00.841405 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:00.842660 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:00.843131 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:00.843231 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:00.847219 (+ 3988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.130:38981: BlockingRecv error: recv got EOF from 127.18.159.130:38981 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 02:08:01 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:01 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:01.342341 28319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:01.326254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.131:59311)
0926 02:08:01.326953 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:01.326974 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:01.327148 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:01.327526 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:01.327531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:01.328016 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:01.328238 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:01.328248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:01.329327 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:01.329334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:01.330505 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:01.330538 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:01.331679 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:01.331707 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:01.331943 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:01.338643 (+ 6700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:01.338670 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:01.342054 (+ 3384us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":382,"threads_started":1}
W20260926 02:08:01.342626 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:01.326474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:59311 (local address 127.18.159.190:42003)
0926 02:08:01.326795 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:01.326801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:01.327073 (+ 272us) server_negotiation.cc:407] Connection header received
0926 02:08:01.327304 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:01.327309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:01.327383 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:01.327506 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:01.328403 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:01.329166 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:01.330699 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:01.331359 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:01.331513 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:01.342345 (+ 10832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.131:59311: BlockingRecv error: recv got EOF from 127.18.159.131:59311 (error 108)
Metrics: {"server-negotiator.queue_time_us":174}
Sep 26 02:08:01 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:01 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:01.367061 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:01.358240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.129:35149)
0926 02:08:01.358969 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:01.358989 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:01.359114 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:01.359438 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:01.359441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:01.359765 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:01.359935 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:01.359944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:01.361073 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:01.361078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:01.361996 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:01.362009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:01.362688 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:01.362692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:01.362820 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:01.363482 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:01.363502 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:01.366684 (+ 3182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":481,"threads_started":1}
W20260926 02:08:01.367462 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:01.358379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:35149 (local address 127.18.159.190:42003)
0926 02:08:01.358581 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:01.358587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:01.359111 (+ 524us) server_negotiation.cc:407] Connection header received
0926 02:08:01.359202 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:01.359205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:01.359279 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:01.359378 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:01.360093 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:01.360922 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:01.362147 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:01.362551 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:01.362650 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:01.367153 (+ 4503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.129:35149: BlockingRecv error: recv got EOF from 127.18.159.129:35149 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 02:08:01 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:01 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:01.857494 28321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:01.848458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.130:47233)
0926 02:08:01.848983 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:01.849000 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:01.849130 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:01.849506 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:01.849510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:01.849828 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:01.850005 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:01.850014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:01.851145 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:01.851149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:01.852123 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:01.852135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:01.852869 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:01.852874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:01.853043 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:01.853779 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:01.853797 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:01.857113 (+ 3316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":322,"threads_started":1}
W20260926 02:08:01.857542 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:01.848541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:47233 (local address 127.18.159.190:42003)
0926 02:08:01.848797 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:01.848806 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:01.849148 (+ 342us) server_negotiation.cc:407] Connection header received
0926 02:08:01.849258 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:01.849262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:01.849329 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:01.849417 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:01.850144 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:01.850983 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:01.852297 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:01.852713 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:01.852818 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:01.857311 (+ 4493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.130:47233: BlockingRecv error: recv got EOF from 127.18.159.130:47233 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 02:08:02 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:02 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:02.352305 28322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:02.343490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.131:48495)
0926 02:08:02.344024 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:02.344056 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:02.344246 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:02.344653 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:02.344657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:02.344993 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:02.345156 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:02.345164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:02.346333 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:02.346337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:02.347388 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:02.347402 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:02.348391 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:02.348395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:02.348540 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:02.349213 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:02.349231 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:02.351986 (+ 2755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":309,"threads_started":1}
W20260926 02:08:02.352576 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:02.343702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:48495 (local address 127.18.159.190:42003)
0926 02:08:02.343886 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:02.343890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:02.344202 (+ 312us) server_negotiation.cc:407] Connection header received
0926 02:08:02.344410 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:02.344419 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:02.344506 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:02.344602 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:02.345330 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:02.346149 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:02.347537 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:02.348270 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:02.348376 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:02.352291 (+ 3915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.131:48495: BlockingRecv error: recv got EOF from 127.18.159.131:48495 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 02:08:02 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:02 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:02.378281 28323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:02.368412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.129:40085)
0926 02:08:02.369167 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:02.369187 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:02.369405 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:02.369785 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:02.369790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:02.370180 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:02.370392 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:02.370404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:02.371874 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:02.371883 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:02.373324 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:02.373343 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:02.374107 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:02.374113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:02.374315 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:02.374993 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:02.375011 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:02.377946 (+ 2935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":476,"threads_started":1}
W20260926 02:08:02.378468 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:02.368486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:40085 (local address 127.18.159.190:42003)
0926 02:08:02.368739 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:02.368745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:02.369279 (+ 534us) server_negotiation.cc:407] Connection header received
0926 02:08:02.369569 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:02.369578 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:02.369659 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:02.369756 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:02.370587 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:02.371750 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:02.373494 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:02.373953 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:02.374094 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:02.378189 (+ 4095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.129:40085: BlockingRecv error: recv got EOF from 127.18.159.129:40085 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 26 02:08:02 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:02 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:02.867568 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:02.858552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.130:46411)
0926 02:08:02.859044 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:02.859060 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:02.859234 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:02.859595 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:02.859600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:02.860009 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:02.860198 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:02.860207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:02.861528 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:02.861536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:02.862788 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:02.862802 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:02.863505 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:02.863511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:02.863705 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:02.864420 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:02.864444 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:02.867228 (+ 2784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":275,"threads_started":1}
W20260926 02:08:02.867668 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:02.858617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:46411 (local address 127.18.159.190:42003)
0926 02:08:02.858861 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:02.858870 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:02.859159 (+ 289us) server_negotiation.cc:407] Connection header received
0926 02:08:02.859379 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:02.859383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:02.859456 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:02.859565 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:02.860358 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:02.861390 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:02.862954 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:02.863348 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:02.863490 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:02.867431 (+ 3941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.130:46411: BlockingRecv error: recv got EOF from 127.18.159.130:46411 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 02:08:03 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:03 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:03.361887 28325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:03.353269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.131:51639)
0926 02:08:03.353778 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:03.353797 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:03.353958 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:03.354368 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:03.354373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:03.354771 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:03.354920 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:03.354929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:03.355922 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:03.355929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:03.356959 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:03.357028 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:03.357777 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:03.357783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:03.357961 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:03.358665 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:03.358683 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:03.361544 (+ 2861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":336,"threads_started":1}
W20260926 02:08:03.362033 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:03.353282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:51639 (local address 127.18.159.190:42003)
0926 02:08:03.353482 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:03.353491 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:03.353889 (+ 398us) server_negotiation.cc:407] Connection header received
0926 02:08:03.354083 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:03.354096 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:03.354225 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:03.354332 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:03.355078 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:03.355779 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:03.357175 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:03.357622 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:03.357760 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:03.361753 (+ 3993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.131:51639: BlockingRecv error: recv got EOF from 127.18.159.131:51639 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 02:08:03 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:03 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:03.388612 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:03.379428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.129:41507)
0926 02:08:03.379971 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:03.379988 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:03.380108 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:03.380567 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:03.380571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:03.380994 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:03.381160 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:03.381169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:03.382461 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:03.382465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:03.383450 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:03.383465 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:03.384322 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:03.384329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:03.384491 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:03.385129 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:03.385149 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:03.388269 (+ 3120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":268,"threads_started":1}
W20260926 02:08:03.388842 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:03.379494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:41507 (local address 127.18.159.190:42003)
0926 02:08:03.379723 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:03.379731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:03.380075 (+ 344us) server_negotiation.cc:407] Connection header received
0926 02:08:03.380353 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:03.380357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:03.380421 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:03.380510 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:03.381323 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:03.382260 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:03.383637 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:03.384060 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:03.384340 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:03.388567 (+ 4227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.129:41507: BlockingRecv error: recv got EOF from 127.18.159.129:41507 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 02:08:03 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:03 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:03.877367 28327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:03.868760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.130:39843)
0926 02:08:03.869330 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:03.869346 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:03.869473 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:03.869759 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:03.869762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:03.870113 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:03.870267 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:03.870275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:03.871391 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:03.871395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:03.872449 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:03.872461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:03.873182 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:03.873186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:03.873321 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:03.873959 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:03.873975 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:03.877000 (+ 3025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":318,"threads_started":1}
W20260926 02:08:03.877503 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:03.868954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:39843 (local address 127.18.159.190:42003)
0926 02:08:03.869123 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:03.869127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:03.869485 (+ 358us) server_negotiation.cc:407] Connection header received
0926 02:08:03.869567 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:03.869571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:03.869631 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:03.869720 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:03.870414 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:03.871227 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:03.872600 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:03.873037 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:03.873149 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:03.877246 (+ 4097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.130:39843: BlockingRecv error: recv got EOF from 127.18.159.130:39843 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 02:08:04 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:04 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:04.375244 28328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:04.363100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.131:40671)
0926 02:08:04.363636 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:04.363656 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:04.363767 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:04.364124 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:04.364127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:04.364521 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:04.364676 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:04.364685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:04.366009 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:04.366018 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:04.367798 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:04.367820 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:04.369083 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:04.369093 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:04.369316 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:04.370444 (+ 1128us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:04.370470 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:04.374772 (+ 4302us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":261,"threads_started":1}
W20260926 02:08:04.375504 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:04.363149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:40671 (local address 127.18.159.190:42003)
0926 02:08:04.363325 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:04.363330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:04.363782 (+ 452us) server_negotiation.cc:407] Connection header received
0926 02:08:04.363861 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:04.363865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:04.363941 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:04.364017 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:04.364832 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:04.365818 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:04.368031 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:04.368871 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:04.369056 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:04.375138 (+ 6082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.131:40671: BlockingRecv error: recv got EOF from 127.18.159.131:40671 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 02:08:04 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:04 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:04.398080 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:04.389691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.129:58267)
0926 02:08:04.390204 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:04.390221 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:04.390347 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:04.390635 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:04.390639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:04.391019 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:04.391167 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:04.391176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:04.392352 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:04.392361 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:04.393398 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:04.393411 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:04.394190 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:04.394195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:04.394324 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:04.394966 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:04.394984 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:04.397869 (+ 2885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":290,"threads_started":1}
W20260926 02:08:04.398406 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:04.389783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:58267 (local address 127.18.159.190:42003)
0926 02:08:04.390077 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:04.390083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:04.390332 (+ 249us) server_negotiation.cc:407] Connection header received
0926 02:08:04.390449 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:04.390453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:04.390514 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:04.390597 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:04.391334 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:04.392088 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:04.393545 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:04.394016 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:04.394116 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:04.398145 (+ 4029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.129:58267: BlockingRecv error: recv got EOF from 127.18.159.129:58267 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 26 02:08:04 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:04 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:04.891017 28330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:04.878421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.130:51233)
0926 02:08:04.878981 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:04.878997 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:04.879110 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:04.879401 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:04.879404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:04.879763 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:04.879917 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:04.879925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:04.880989 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:04.880993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:04.881907 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:04.881918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:04.882700 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:04.882705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:04.882827 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:04.887346 (+ 4519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:04.887368 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:04.890786 (+ 3418us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":312,"threads_started":1}
W20260926 02:08:04.891201 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:04.878468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:51233 (local address 127.18.159.190:42003)
0926 02:08:04.878637 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:04.878641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:04.879073 (+ 432us) server_negotiation.cc:407] Connection header received
0926 02:08:04.879213 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:04.879216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:04.879274 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:04.879359 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:04.880094 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:04.880844 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:04.882073 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:04.882561 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:04.882660 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:04.890991 (+ 8331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.130:51233: BlockingRecv error: recv got EOF from 127.18.159.130:51233 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 02:08:05 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:05 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:05.386034 28331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:05.376465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.131:57845)
0926 02:08:05.377247 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:05.377267 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:05.377429 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:05.377771 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:05.377776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:05.378268 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:05.378477 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:05.378489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:05.379598 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:05.379604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:05.380572 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:05.380585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:05.381364 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:05.381370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:05.381558 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:05.382199 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:05.382217 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:05.385672 (+ 3455us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":552,"threads_started":1}
W20260926 02:08:05.386221 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:05.376542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:57845 (local address 127.18.159.190:42003)
0926 02:08:05.376732 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:05.376737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:05.377357 (+ 620us) server_negotiation.cc:407] Connection header received
0926 02:08:05.377554 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:05.377560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:05.377633 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:05.377739 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:05.378620 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:05.379430 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:05.380737 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:05.381183 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:05.381321 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:05.385915 (+ 4594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.131:57845: BlockingRecv error: recv got EOF from 127.18.159.131:57845 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 02:08:05 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:05 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:05.407866 28332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:05.399252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.129:47089)
0926 02:08:05.399773 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:05.399789 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:05.399964 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:05.400387 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:05.400393 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:05.400830 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:05.401026 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:05.401037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:05.402129 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:05.402139 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:05.403068 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:05.403080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:05.403829 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:05.403834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:05.404006 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:05.404630 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:05.404666 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:05.407612 (+ 2946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":239,"threads_started":1}
W20260926 02:08:05.408030 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:05.399309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:47089 (local address 127.18.159.190:42003)
0926 02:08:05.399545 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:05.399553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:05.399888 (+ 335us) server_negotiation.cc:407] Connection header received
0926 02:08:05.400114 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:05.400118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:05.400247 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:05.400353 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:05.401182 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:05.401971 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:05.403218 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:05.403666 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:05.403813 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:05.407822 (+ 4009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.129:47089: BlockingRecv error: recv got EOF from 127.18.159.129:47089 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 02:08:05 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:05 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:05.900750 28333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:05.892136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.130:57233)
0926 02:08:05.892734 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:05.892751 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:05.892868 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:05.893174 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:05.893177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:05.893574 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:05.893734 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:05.893742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:05.894713 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:05.894716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:05.895654 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:05.895665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:05.896439 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:05.896445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:05.896574 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:05.897139 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:05.897156 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:05.900496 (+ 3340us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":259,"threads_started":1}
W20260926 02:08:05.900910 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:05.892246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:57233 (local address 127.18.159.190:42003)
0926 02:08:05.892484 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:05.892494 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:05.892842 (+ 348us) server_negotiation.cc:407] Connection header received
0926 02:08:05.892980 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:05.892984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:05.893045 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:05.893130 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:05.893861 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:05.894588 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:05.895781 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:05.896271 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:05.896413 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:05.900688 (+ 4275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.130:57233: BlockingRecv error: recv got EOF from 127.18.159.130:57233 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 02:08:06 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:06 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:06.405843 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:06.387403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.131:49539)
0926 02:08:06.388257 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:06.388281 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:06.388432 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:06.388737 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:06.388741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:06.389117 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:06.389274 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:06.389282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:06.390240 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:06.390243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:06.391197 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:06.391208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:06.391891 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:06.391896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:06.392013 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:06.402074 (+ 10061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:06.402096 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:06.405543 (+ 3447us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":431,"threads_started":1}
W20260926 02:08:06.406062 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:06.387525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:49539 (local address 127.18.159.190:42003)
0926 02:08:06.387731 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:06.387737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:06.388369 (+ 632us) server_negotiation.cc:407] Connection header received
0926 02:08:06.388549 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:06.388552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:06.388614 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:06.388695 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:06.389399 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:06.390120 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:06.391337 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:06.391749 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:06.391845 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:06.405816 (+ 13971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.131:49539: BlockingRecv error: recv got EOF from 127.18.159.131:49539 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 02:08:06 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:06 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:06.417292 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:06.408903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.129:48295)
0926 02:08:06.409411 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:06.409427 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:06.409556 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:06.409839 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:06.409842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:06.410186 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:06.410339 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:06.410347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:06.411631 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:06.411636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:06.412832 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:06.412845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:06.413575 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:06.413580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:06.413694 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:06.414281 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:06.414298 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:06.417060 (+ 2762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":277,"threads_started":1}
W20260926 02:08:06.417521 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:06.409157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:48295 (local address 127.18.159.190:42003)
0926 02:08:06.409344 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:06.409350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:06.409515 (+ 165us) server_negotiation.cc:407] Connection header received
0926 02:08:06.409655 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:06.409659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:06.409717 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:06.409804 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:06.410505 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:06.411473 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:06.412972 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:06.413396 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:06.413492 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:06.417282 (+ 3790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.129:48295: BlockingRecv error: recv got EOF from 127.18.159.129:48295 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 02:08:06 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:06 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:06.911123 28336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:06.901839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.130:56085)
0926 02:08:06.902427 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:06.902444 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:06.902558 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:06.902894 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:06.902899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:06.903319 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:06.903526 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:06.903537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:06.904966 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:06.904971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:06.906068 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:06.906120 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:06.907159 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:06.907165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:06.907300 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:06.907923 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:06.907941 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:06.910892 (+ 2951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":291,"threads_started":1}
W20260926 02:08:06.911389 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:06.901937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:56085 (local address 127.18.159.190:42003)
0926 02:08:06.902146 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:06.902151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:06.902524 (+ 373us) server_negotiation.cc:407] Connection header received
0926 02:08:06.902664 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:06.902668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:06.902732 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:06.902839 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:06.903684 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:06.904722 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:06.906367 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:06.906937 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:06.907060 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:06.911111 (+ 4051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.130:56085: BlockingRecv error: recv got EOF from 127.18.159.130:56085 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 26 02:08:07 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:07 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:07.416059 28337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:07.406907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.131:38125)
0926 02:08:07.407460 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:07.407482 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:07.407651 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:07.408009 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:07.408015 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:07.408536 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:07.408733 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:07.408746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:07.410002 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:07.410009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:07.410959 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:07.410971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:07.411918 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:07.411925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:07.412124 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:07.412860 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:07.412877 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:07.415717 (+ 2840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":275,"threads_started":1}
W20260926 02:08:07.416240 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:07.407093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:38125 (local address 127.18.159.190:42003)
0926 02:08:07.407343 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:07.407349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:07.407578 (+ 229us) server_negotiation.cc:407] Connection header received
0926 02:08:07.407783 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:07.407789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:07.407858 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:07.407971 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:07.408869 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:07.409858 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:07.411129 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:07.411732 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:07.411883 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:07.415942 (+ 4059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.131:38125: BlockingRecv error: recv got EOF from 127.18.159.131:38125 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 26 02:08:07 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:07 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:07.428073 28338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:07.418298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.129:35611)
0926 02:08:07.418849 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:07.418865 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:07.419050 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:07.419375 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:07.419379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:07.419787 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:07.419956 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:07.419965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:07.421215 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:07.421219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:07.422371 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:07.422385 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:07.423274 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:07.423282 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:07.423558 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:07.424274 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:07.424292 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:07.427701 (+ 3409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":286,"threads_started":1}
W20260926 02:08:07.428210 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:07.418394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:35611 (local address 127.18.159.190:42003)
0926 02:08:07.418649 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:07.418656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:07.418977 (+ 321us) server_negotiation.cc:407] Connection header received
0926 02:08:07.419189 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:07.419192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:07.419251 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:07.419349 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:07.420100 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:07.421051 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:07.422504 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:07.423091 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:07.423238 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:07.427929 (+ 4691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.129:35611: BlockingRecv error: recv got EOF from 127.18.159.129:35611 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 26 02:08:07 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:07 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:07.920746 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:07.912269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.130:54105)
0926 02:08:07.912836 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:07.912853 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:07.912970 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:07.913281 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:07.913284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:07.913657 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:07.913814 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:07.913822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:07.914830 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:07.914834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:07.915772 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:07.915783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:07.916562 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:07.916566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:07.916690 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:07.917251 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:07.917269 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:07.920438 (+ 3169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":367,"threads_started":1}
W20260926 02:08:07.920970 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:07.912367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:54105 (local address 127.18.159.190:42003)
0926 02:08:07.912628 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:07.912634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:07.912953 (+ 319us) server_negotiation.cc:407] Connection header received
0926 02:08:07.913092 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:07.913096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:07.913157 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:07.913239 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:07.913939 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:07.914713 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:07.915907 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:07.916417 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:07.916558 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:07.920671 (+ 4113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.130:54105: BlockingRecv error: recv got EOF from 127.18.159.130:54105 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 26 02:08:08 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:08 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:08.425977 28340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:08.417179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.131:56185)
0926 02:08:08.417722 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:08.417742 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:08.417912 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:08.418312 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:08.418316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:08.418729 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:08.418906 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:08.418915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:08.419911 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:08.419920 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:08.421070 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:08.421083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:08.421917 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:08.421923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:08.422054 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:08.422689 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:08.422707 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:08.425734 (+ 3027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":294,"threads_started":1}
W20260926 02:08:08.426218 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:08.417240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:56185 (local address 127.18.159.190:42003)
0926 02:08:08.417482 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:08.417518 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:08.417846 (+ 328us) server_negotiation.cc:407] Connection header received
0926 02:08:08.418107 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:08.418112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:08.418172 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:08.418282 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:08.419067 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:08.419770 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:08.421219 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:08.421741 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:08.421897 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:08.425958 (+ 4061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.131:56185: BlockingRecv error: recv got EOF from 127.18.159.131:56185 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 26 02:08:08 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:08 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:08.437589 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:08.429125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.129:49399)
0926 02:08:08.429630 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:08.429646 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:08.429757 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:08.430095 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:08.430102 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:08.430450 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:08.430611 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:08.430619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:08.431678 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:08.431682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:08.432895 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:08.432907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:08.433860 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:08.433864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:08.433977 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:08.434579 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:08.434596 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:08.437347 (+ 2751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":282,"threads_started":1}
W20260926 02:08:08.437793 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:08.429184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:49399 (local address 127.18.159.190:42003)
0926 02:08:08.429352 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:08.429357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:08.429718 (+ 361us) server_negotiation.cc:407] Connection header received
0926 02:08:08.429858 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:08.429861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:08.429924 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:08.430012 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:08.430775 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:08.431559 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:08.433034 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:08.433695 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:08.433817 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:08.437559 (+ 3742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.129:49399: BlockingRecv error: recv got EOF from 127.18.159.129:49399 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 02:08:08 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:08 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:08.930159 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:08.921833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.130:54731)
0926 02:08:08.922373 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:08.922389 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:08.922548 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:08.922891 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:08.922895 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:08.923370 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:08.923527 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:08.923536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:08.924589 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:08.924596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:08.925526 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:08.925537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:08.926271 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:08.926277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:08.926431 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:08.927069 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:08.927087 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:08.929836 (+ 2749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":279,"threads_started":1}
W20260926 02:08:08.930282 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:08.921891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:54731 (local address 127.18.159.190:42003)
0926 02:08:08.922083 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:08.922090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:08.922478 (+ 388us) server_negotiation.cc:407] Connection header received
0926 02:08:08.922674 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:08.922680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:08.922756 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:08.922861 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:08.923672 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:08.924455 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:08.925666 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:08.926086 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:08.926220 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:08.930034 (+ 3814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.130:54731: BlockingRecv error: recv got EOF from 127.18.159.130:54731 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 02:08:09 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:09 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:09.436733 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:09.427128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.131:45313)
0926 02:08:09.427683 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:09.427704 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:09.427870 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:09.428293 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:09.428297 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:09.428745 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:09.428886 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:09.428895 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:09.429973 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:09.429980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:09.431469 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:09.431482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:09.432257 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:09.432264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:09.432440 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:09.433188 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:09.433210 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:09.436494 (+ 3284us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":323,"threads_started":1}
W20260926 02:08:09.437142 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:09.427379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:45313 (local address 127.18.159.190:42003)
0926 02:08:09.427618 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:09.427625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:09.427798 (+ 173us) server_negotiation.cc:407] Connection header received
0926 02:08:09.428035 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:09.428042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:09.428110 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:09.428267 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:09.429073 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:09.429824 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:09.431640 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:09.432056 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:09.432213 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:09.436737 (+ 4524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.131:45313: BlockingRecv error: recv got EOF from 127.18.159.131:45313 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 26 02:08:09 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:09 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:09.447139 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:09.438541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.129:33199)
0926 02:08:09.439056 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:09.439072 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:09.439232 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:09.439561 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:09.439565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:09.439961 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:09.440142 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:09.440201 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:09.441226 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:09.441233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:09.442137 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:09.442148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:09.442890 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:09.442895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:09.443078 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:09.443704 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:09.443721 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:09.446827 (+ 3106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":269,"threads_started":1}
W20260926 02:08:09.447322 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:09.438619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:33199 (local address 127.18.159.190:42003)
0926 02:08:09.438882 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:09.438890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:09.439167 (+ 277us) server_negotiation.cc:407] Connection header received
0926 02:08:09.439363 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:09.439368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:09.439424 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:09.439535 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:09.440339 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:09.441081 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:09.442285 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:09.442727 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:09.442860 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:09.447052 (+ 4192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.129:33199: BlockingRecv error: recv got EOF from 127.18.159.129:33199 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 26 02:08:09 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:09 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:09.940150 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:09.931282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.130:39671)
0926 02:08:09.931824 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:09.931845 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:09.931979 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:09.932461 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:09.932465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:09.933033 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:09.933254 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:09.933276 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:09.934551 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:09.934555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:09.935483 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:09.935496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:09.936293 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:09.936297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:09.936437 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:09.937137 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:09.937156 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:09.939905 (+ 2749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":314,"threads_started":1}
W20260926 02:08:09.940510 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:09.931423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:39671 (local address 127.18.159.190:42003)
0926 02:08:09.931605 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:09.931610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:09.931949 (+ 339us) server_negotiation.cc:407] Connection header received
0926 02:08:09.932127 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:09.932131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:09.932274 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:09.932419 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:09.933428 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:09.934406 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:09.935630 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:09.936074 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:09.936254 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:09.940198 (+ 3944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.130:39671: BlockingRecv error: recv got EOF from 127.18.159.130:39671 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 02:08:10 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:10 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:10.445861 28346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:10.437694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.131:51229)
0926 02:08:10.438204 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:10.438226 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:10.438333 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:10.438598 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:10.438601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:10.439002 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:10.439174 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:10.439182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:10.440116 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:10.440119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:10.441156 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:10.441168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:10.441808 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:10.441813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:10.441934 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:10.442578 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:10.442595 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:10.445477 (+ 2882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":285,"threads_started":1}
W20260926 02:08:10.445909 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:10.437774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:51229 (local address 127.18.159.190:42003)
0926 02:08:10.437940 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:10.437952 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:10.438302 (+ 350us) server_negotiation.cc:407] Connection header received
0926 02:08:10.438415 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:10.438418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:10.438477 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:10.438558 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:10.439310 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:10.439990 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:10.441280 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:10.441671 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:10.441768 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:10.445686 (+ 3918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.131:51229: BlockingRecv error: recv got EOF from 127.18.159.131:51229 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 02:08:10 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:10 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:10.456626 28347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:10.448078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.129:58351)
0926 02:08:10.448612 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:10.448626 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:10.448773 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:10.449173 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:10.449181 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:10.449706 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:10.449921 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:10.449937 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:10.451063 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:10.451068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:10.451962 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:10.451973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:10.452821 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:10.452826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:10.452954 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:10.453599 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:10.453622 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:10.456394 (+ 2772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":313,"threads_started":1}
W20260926 02:08:10.456854 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:10.448218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:58351 (local address 127.18.159.190:42003)
0926 02:08:10.448402 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:10.448407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:10.448706 (+ 299us) server_negotiation.cc:407] Connection header received
0926 02:08:10.448892 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:10.448897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:10.449013 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:10.449110 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:10.450095 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:10.450925 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:10.452131 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:10.452692 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:10.452789 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:10.456599 (+ 3810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.129:58351: BlockingRecv error: recv got EOF from 127.18.159.129:58351 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 02:08:10 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:10 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:10.950271 28348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:10.941250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.130:32819)
0926 02:08:10.941756 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:10.941773 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:10.941903 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:10.942275 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:10.942278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:10.942671 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:10.942834 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:10.942842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:10.944018 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:10.944023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:10.945098 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:10.945112 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:10.946172 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:10.946179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:10.946408 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:10.947115 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:10.947134 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:10.949928 (+ 2794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":308,"threads_started":1}
W20260926 02:08:10.950428 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:10.941376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:32819 (local address 127.18.159.190:42003)
0926 02:08:10.941662 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:10.941669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:10.941872 (+ 203us) server_negotiation.cc:407] Connection header received
0926 02:08:10.942049 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:10.942053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:10.942119 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:10.942208 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:10.943026 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:10.943844 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:10.945314 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:10.945958 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:10.946119 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:10.950178 (+ 4059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.130:32819: BlockingRecv error: recv got EOF from 127.18.159.130:32819 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Sep 26 02:08:11 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:11 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 02:08:11.459980 28312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:11.446957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:60591 (local address 127.18.159.190:42003)
0926 02:08:11.447147 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:11.447151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:11.447556 (+ 405us) server_negotiation.cc:407] Connection header received
0926 02:08:11.447687 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:11.447691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:11.447759 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:11.447837 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:11.448606 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:11.449316 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:11.450458 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:11.450884 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:11.450975 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:11.459672 (+ 8697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.131:60591: BlockingRecv error: recv got EOF from 127.18.159.131:60591 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 02:08:11.460548 28349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:11.446923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.131:60591)
0926 02:08:11.447458 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:11.447480 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:11.447591 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:11.447915 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:11.447918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:11.448333 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:11.448487 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:11.448496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:11.449438 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:11.449442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:11.450341 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:11.450351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:11.451009 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:11.451014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:11.451142 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:11.455845 (+ 4703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:11.455899 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:11.460223 (+ 4324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":307,"threads_started":1}
Sep 26 02:08:11 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:11 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:11.466634 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:11.457668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.129:37401)
0926 02:08:11.459130 (+ 1462us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:11.459151 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:11.459292 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:11.459676 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:11.459680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:11.460005 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:11.460272 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:11.460292 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:11.461492 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:11.461496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:11.462422 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:11.462433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:11.463123 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:11.463127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:11.463238 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:11.463794 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:11.463811 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:11.466417 (+ 2606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1299,"thread_start_us":1166,"threads_started":1}
W20260926 02:08:11.467010 28351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:11.457752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:37401 (local address 127.18.159.190:42003)
0926 02:08:11.458638 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:11.458647 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:11.459248 (+ 601us) server_negotiation.cc:407] Connection header received
0926 02:08:11.459453 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:11.459457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:11.459528 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:11.459634 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:11.460417 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:11.461360 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:11.462557 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:11.462960 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:11.463051 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:11.466648 (+ 3597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.129:37401: BlockingRecv error: recv got EOF from 127.18.159.129:37401 (error 108)
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":560,"threads_started":1}
Sep 26 02:08:11 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:11 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:11.960311 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:11.951333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.130:57955)
0926 02:08:11.951901 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:11.951918 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:11.952040 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:11.952375 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:11.952378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:11.952753 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:11.952905 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:11.952914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:11.954006 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:11.954010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:11.954978 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:11.954990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:11.955979 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:11.955987 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:11.956159 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:11.956946 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:11.956969 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:11.959943 (+ 2974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":302,"threads_started":1}
W20260926 02:08:11.960484 28351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:11.951539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:57955 (local address 127.18.159.190:42003)
0926 02:08:11.951734 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:11.951739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:11.951998 (+ 259us) server_negotiation.cc:407] Connection header received
0926 02:08:11.952150 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:11.952176 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:11.952238 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:11.952331 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:11.953074 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:11.953796 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:11.955200 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:11.955751 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:11.955892 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:11.960213 (+ 4321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.130:57955: BlockingRecv error: recv got EOF from 127.18.159.130:57955 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 02:08:12 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:12 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:12.473222 28353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:12.461708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.131:40521)
0926 02:08:12.462372 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:12.462398 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:12.462540 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:12.463099 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:12.463104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:12.463559 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:12.463753 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:12.463762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:12.465578 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:12.465582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:12.466512 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:12.466524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:12.468524 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:12.468535 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:12.468804 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:12.469543 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:12.469568 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:12.472841 (+ 3273us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":285,"threads_started":1}
W20260926 02:08:12.473528 28354 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:12.461810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:40521 (local address 127.18.159.190:42003)
0926 02:08:12.462616 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:12.462625 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:12.462651 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:08:12.462761 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:12.462769 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:12.462886 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:12.463017 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:12.463932 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:12.465410 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:12.466672 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:12.467633 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:12.467800 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:12.473105 (+ 5305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.131:40521: BlockingRecv error: recv got EOF from 127.18.159.131:40521 (error 108)
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":544,"threads_started":1}
Sep 26 02:08:12 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:12 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:12.479635 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:12.467506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.129:49927)
0926 02:08:12.468444 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:12.468471 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:12.468626 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:12.469605 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:12.469610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:12.470077 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:12.470297 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:12.470309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:12.472243 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:12.472249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:12.473985 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:12.474005 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:12.474831 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:12.474837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:12.475030 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:12.475854 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:12.475883 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:12.479273 (+ 3390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":639,"threads_started":1}
W20260926 02:08:12.480006 28356 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:12.467617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:49927 (local address 127.18.159.190:42003)
0926 02:08:12.469250 (+ 1633us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:12.469256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:12.469274 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:08:12.469329 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:12.469336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:12.469420 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:12.469552 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:12.470463 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:12.472050 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:12.474158 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:12.474684 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:12.474817 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:12.479524 (+ 4707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.129:49927: BlockingRecv error: recv got EOF from 127.18.159.129:49927 (error 108)
Metrics: {"server-negotiator.queue_time_us":1441,"thread_start_us":1179,"threads_started":1}
Sep 26 02:08:12 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:12 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:12.971042 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:12.961735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.130:48045)
0926 02:08:12.962236 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:12.962262 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:12.962387 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:12.962706 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:12.962709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:12.963091 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:12.963316 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:12.963325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:12.964695 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:12.964703 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:12.965853 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:12.965874 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:12.966770 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:12.966776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:12.966973 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:12.967746 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:12.967765 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:12.970743 (+ 2978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":278,"threads_started":1}
W20260926 02:08:12.971256 28356 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:12.961864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:48045 (local address 127.18.159.190:42003)
0926 02:08:12.962063 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:12.962068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:12.962400 (+ 332us) server_negotiation.cc:407] Connection header received
0926 02:08:12.962492 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:12.962496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:12.962560 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:12.962654 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:12.963490 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:12.964483 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:12.966003 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:12.966565 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:12.966707 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:12.971000 (+ 4293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.130:48045: BlockingRecv error: recv got EOF from 127.18.159.130:48045 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 02:08:13 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:13 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.131@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:13.483984 28358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:13.474631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.131:50147)
0926 02:08:13.475112 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:13.475135 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:13.475248 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:13.475746 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:13.475750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:13.476107 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:13.476385 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:13.476394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:13.477504 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:13.477509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:13.478436 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:13.478449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:13.479237 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:13.479241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:13.479383 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:13.480127 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:13.480144 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:13.483627 (+ 3483us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":247,"threads_started":1}
W20260926 02:08:13.484143 28359 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:13.474739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:50147 (local address 127.18.159.190:42003)
0926 02:08:13.475459 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:13.475464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:13.475481 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:08:13.475529 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:13.475532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:13.475596 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:13.475687 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:13.476522 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:13.477338 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:13.478596 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:13.479077 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:13.479185 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:13.483846 (+ 4661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.131:50147: BlockingRecv error: recv got EOF from 127.18.159.131:50147 (error 108)
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":518,"threads_started":1}
Sep 26 02:08:13 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:13 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.129@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:13.490381 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:13.480836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.129:34993)
0926 02:08:13.481747 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:13.481764 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:13.481946 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:13.482285 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:13.482291 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:13.482817 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:13.483050 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:13.483063 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:13.484560 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:13.484566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:13.485692 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:13.485706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:13.486438 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:13.486442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:13.486565 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:13.487277 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:13.487296 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:13.490124 (+ 2828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":613,"threads_started":1}
W20260926 02:08:13.490671 28361 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:13.480955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:34993 (local address 127.18.159.190:42003)
0926 02:08:13.481518 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:13.481523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:13.481891 (+ 368us) server_negotiation.cc:407] Connection header received
0926 02:08:13.482074 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:13.482083 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:13.482141 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:13.482252 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:13.483217 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:13.484398 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:13.485833 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:13.486292 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:13.486396 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:13.490398 (+ 4002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.129:34993: BlockingRecv error: recv got EOF from 127.18.159.129:34993 (error 108)
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":263,"threads_started":1}
Sep 26 02:08:13 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:08:13 dist-test-slave-03rn krb5kdc[27308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.159.130@KRBTEST.COM for oryx/127.18.159.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:08:13.982122 28362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:13.972544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:42003 (local address 127.18.159.130:60643)
0926 02:08:13.973194 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:13.973226 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:13.973387 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:13.973843 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:13.973849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:13.974357 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:13.974626 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:13.974639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:13.975876 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:13.975880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:13.976951 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:13.976964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:13.977748 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:13.977753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:13.977869 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:13.978649 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:13.978667 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:13.981718 (+ 3051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.159.190:42003: Server oryx/127.18.159.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":363,"threads_started":1}
W20260926 02:08:13.982228 28361 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:13.972687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:60643 (local address 127.18.159.190:42003)
0926 02:08:13.972891 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:13.972899 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:13.973349 (+ 450us) server_negotiation.cc:407] Connection header received
0926 02:08:13.973539 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:13.973545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:13.973657 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:13.973773 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:13.974796 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:13.975667 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:13.977131 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:13.977609 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:13.977711 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:13.981974 (+ 4263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.159.130:60643: BlockingRecv error: recv got EOF from 127.18.159.130:60643 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:08:14.285243 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 27897
I20260926 02:08:14.298422 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:14.286450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39816 (local address 127.18.159.129:33563)
0926 02:08:14.287031 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:14.287038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:14.287068 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:08:14.287128 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:14.287134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:14.287349 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:14.287525 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:14.289145 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:14.290212 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:14.291474 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:14.292050 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:14.292217 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:14.294878 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:14.294889 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:14.294908 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:08:14.294956 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:14.297867 (+ 2911us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.18.159.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 02:08:14.298157 (+ 290us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39816: Request ticket server kudu/127.18.159.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":292,"threads_started":1}
W20260926 02:08:14.298548 28364 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39816: Request ticket server kudu/127.18.159.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 02:08:14.298616 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 27897: Not authorized: Client connection negotiation failed: client connection to 127.18.159.129:33563: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.18.159.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 02:08:14.298911 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 27897
I20260926 02:08:14.315469 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 28049
I20260926 02:08:14.328665 28365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:14.316670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42456 (local address 127.18.159.130:36229)
0926 02:08:14.317314 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:14.317321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:14.317352 (+ 31us) server_negotiation.cc:407] Connection header received
0926 02:08:14.317414 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:14.317420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:14.317649 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:14.317819 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:14.319122 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:14.320569 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:14.321937 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:14.322536 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:14.322645 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:14.325367 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:14.325376 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:14.325411 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:08:14.325450 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:14.328023 (+ 2573us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.18.159.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 02:08:14.328422 (+ 399us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42456: Request ticket server kudu/127.18.159.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":346,"threads_started":1}
W20260926 02:08:14.328800 28365 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42456: Request ticket server kudu/127.18.159.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 02:08:14.328784 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 28049: Not authorized: Client connection negotiation failed: client connection to 127.18.159.130:36229: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.18.159.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 02:08:14.328938 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 28049
I20260926 02:08:14.345222 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 28181
I20260926 02:08:14.357678 28366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:14.346208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41868 (local address 127.18.159.131:45373)
0926 02:08:14.346831 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:14.346839 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:14.346865 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:08:14.346925 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:14.346932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:14.347158 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:14.347311 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:14.348642 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:14.349689 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:14.350876 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:14.351460 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:14.351558 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:14.354377 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:14.354386 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:14.354400 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:08:14.354435 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:14.357133 (+ 2698us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.18.159.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 02:08:14.357418 (+ 285us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41868: Request ticket server kudu/127.18.159.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":332,"threads_started":1}
W20260926 02:08:14.357791 28366 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41868: Request ticket server kudu/127.18.159.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 02:08:14.357904 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 28181: Not authorized: Client connection negotiation failed: client connection to 127.18.159.131:45373: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.18.159.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 02:08:14.358098 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 28181
I20260926 02:08:14.375613 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 27828
I20260926 02:08:14.377262 27890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:08:14.492748 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 27828
2026-09-26T02:08:14Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21460 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T02:08:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:08:14Z Disabled control of system clock
I20260926 02:08:14.571346 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:46255
--webserver_interface=127.18.159.190
--webserver_port=0
--builtin_ntp_servers=127.18.159.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:46255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 02:08:14.919493 28371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:08:14.919804 28371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:08:14.919962 28371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:08:14.927618 28371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:08:14.927770 28371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:08:14.927824 28371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:08:14.927917 28371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:08:14.927968 28371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:08:14.937584 28371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:46255
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:46255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:08:14.938938 28371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:08:14.940793 28371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:08:14.953815 28376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:14.953783 28377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:14.953986 28379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:08:14.954579 28371 server_base.cc:1034] running on GCE node
I20260926 02:08:14.955425 28371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:08:14.957067 28371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:08:14.958305 28371 hybrid_clock.cc:648] HybridClock initialized: now 1790388494958258 us; error 54 us; skew 500 ppm
I20260926 02:08:14.958689 28371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:08:14.961199 28371 webserver.cc:460] Webserver started at http://127.18.159.190:42403/ using document root <none> and password file <none>
I20260926 02:08:14.961961 28371 fs_manager.cc:359] Metadata directory not provided
I20260926 02:08:14.962098 28371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:08:14.962441 28371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:08:14.965545 28371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "eb8291337c5d48ee979816a2fdc134d2"
format_stamp: "Formatted at 2026-09-26 02:08:14 on dist-test-slave-03rn"
server_key: "7d12cb7d8401075a0adb3207cfed3d3e"
server_key_iv: "e5cb27bbad63dea25936fc6eead3f7dc"
server_key_version: "encryptionkey@0"
I20260926 02:08:14.966377 28371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "eb8291337c5d48ee979816a2fdc134d2"
format_stamp: "Formatted at 2026-09-26 02:08:14 on dist-test-slave-03rn"
server_key: "7d12cb7d8401075a0adb3207cfed3d3e"
server_key_iv: "e5cb27bbad63dea25936fc6eead3f7dc"
server_key_version: "encryptionkey@0"
I20260926 02:08:14.971990 28371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:08:14.975908 28385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:14.977547 28371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:08:14.977813 28371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "eb8291337c5d48ee979816a2fdc134d2"
format_stamp: "Formatted at 2026-09-26 02:08:14 on dist-test-slave-03rn"
server_key: "7d12cb7d8401075a0adb3207cfed3d3e"
server_key_iv: "e5cb27bbad63dea25936fc6eead3f7dc"
server_key_version: "encryptionkey@0"
I20260926 02:08:14.978060 28371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:08:14.993978 28371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:08:14.994902 28371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:08:14.995164 28371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:08:15.014680 28371 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:46255
I20260926 02:08:15.014705 28436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:46255 every 8 connection(s)
I20260926 02:08:15.016104 28371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:08:15.020084 28437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:08:15.025089 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 28371
I20260926 02:08:15.025260 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:08:15.026093 19070 external_mini_cluster.cc:1428] Setting key 5738e157ae2b2d7020f1182de5c71714
I20260926 02:08:15.032045 28437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2: Bootstrap starting.
I20260926 02:08:15.035353 28437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2: Neither blocks nor log segments found. Creating new log.
I20260926 02:08:15.036623 28437 log.cc:824] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2: Log is configured to *not* fsync() on all Append() calls
I20260926 02:08:15.040473 28437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2: No bootstrap required, opened a new log
I20260926 02:08:15.041178 28440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:15.028332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47030 (local address 127.18.159.190:46255)
0926 02:08:15.029507 (+ 1175us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:15.029518 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:15.029562 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:08:15.031285 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:15.031310 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:15.031827 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:15.032317 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:15.032950 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:15.034305 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:15.035742 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:15.036589 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:15.036806 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:15.038040 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:15.038050 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:15.038072 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:08:15.038883 (+ 811us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:15.038995 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:08:15.039024 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:15.039155 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:15.039380 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 02:08:15.039767 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":399,"threads_started":1}
I20260926 02:08:15.048322 28437 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8291337c5d48ee979816a2fdc134d2" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 46255 } }
I20260926 02:08:15.048707 28437 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:08:15.048789 28437 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb8291337c5d48ee979816a2fdc134d2, State: Initialized, Role: FOLLOWER
I20260926 02:08:15.049305 28437 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8291337c5d48ee979816a2fdc134d2" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 46255 } }
I20260926 02:08:15.049530 28437 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:08:15.049669 28437 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:08:15.049852 28437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:08:15.052239 28437 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8291337c5d48ee979816a2fdc134d2" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 46255 } }
I20260926 02:08:15.052786 28437 leader_election.cc:302] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb8291337c5d48ee979816a2fdc134d2; no voters:
I20260926 02:08:15.053390 28437 leader_election.cc:288] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:08:15.053572 28442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:08:15.054316 28442 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2 [term 1 LEADER]: Becoming Leader. State: Replica: eb8291337c5d48ee979816a2fdc134d2, State: Running, Role: LEADER
I20260926 02:08:15.054930 28442 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8291337c5d48ee979816a2fdc134d2" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 46255 } }
I20260926 02:08:15.055702 28437 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:08:15.059494 28444 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb8291337c5d48ee979816a2fdc134d2. Latest consensus state: current_term: 1 leader_uuid: "eb8291337c5d48ee979816a2fdc134d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8291337c5d48ee979816a2fdc134d2" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 46255 } } }
I20260926 02:08:15.060452 28444 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2 [sys.catalog]: This master's current role is: LEADER
I20260926 02:08:15.060405 28443 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb8291337c5d48ee979816a2fdc134d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8291337c5d48ee979816a2fdc134d2" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 46255 } } }
I20260926 02:08:15.060953 28443 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2 [sys.catalog]: This master's current role is: LEADER
I20260926 02:08:15.063365 28449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:08:15.077458 28449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:08:15.091791 28449 catalog_manager.cc:1347] Generated new cluster ID: f9b23d56e14e458b97cbf5c5b56b9d00
I20260926 02:08:15.092015 28449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:08:15.116314 28449 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:08:15.118275 28449 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:08:15.132565 28449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb8291337c5d48ee979816a2fdc134d2: Generated new TSK 0
I20260926 02:08:15.133757 28449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:08:15.144774 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:46255
--builtin_ntp_servers=127.18.159.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 02:08:15.486048 28461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:08:15.486358 28461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:08:15.486541 28461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:08:15.494025 28461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:08:15.494146 28461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:08:15.494328 28461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:08:15.503692 28461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:46255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:08:15.505234 28461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:08:15.506965 28461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:08:15.520749 28466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:15.520712 28469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:15.521134 28467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:08:15.521453 28461 server_base.cc:1034] running on GCE node
I20260926 02:08:15.522150 28461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:08:15.523154 28461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:08:15.524386 28461 hybrid_clock.cc:648] HybridClock initialized: now 1790388495524314 us; error 70 us; skew 500 ppm
I20260926 02:08:15.524762 28461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:08:15.527061 28461 webserver.cc:460] Webserver started at http://127.18.159.129:37605/ using document root <none> and password file <none>
I20260926 02:08:15.527797 28461 fs_manager.cc:359] Metadata directory not provided
I20260926 02:08:15.527932 28461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:08:15.528333 28461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:08:15.531194 28461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "7b48fb314eb5449f90c46827f3bf8889"
format_stamp: "Formatted at 2026-09-26 02:08:15 on dist-test-slave-03rn"
server_key: "cf08cb55b183f12922af3e041140e58a"
server_key_iv: "fa8915a7821c9c95a47c03ceff57d8e7"
server_key_version: "encryptionkey@0"
I20260926 02:08:15.532011 28461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "7b48fb314eb5449f90c46827f3bf8889"
format_stamp: "Formatted at 2026-09-26 02:08:15 on dist-test-slave-03rn"
server_key: "cf08cb55b183f12922af3e041140e58a"
server_key_iv: "fa8915a7821c9c95a47c03ceff57d8e7"
server_key_version: "encryptionkey@0"
I20260926 02:08:15.537338 28461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 02:08:15.541204 28475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:15.542644 28461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:08:15.542829 28461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "7b48fb314eb5449f90c46827f3bf8889"
format_stamp: "Formatted at 2026-09-26 02:08:15 on dist-test-slave-03rn"
server_key: "cf08cb55b183f12922af3e041140e58a"
server_key_iv: "fa8915a7821c9c95a47c03ceff57d8e7"
server_key_version: "encryptionkey@0"
I20260926 02:08:15.543067 28461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:08:15.564091 28461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:08:15.565061 28461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:08:15.565390 28461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:08:15.566373 28461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:08:15.567883 28461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:08:15.568010 28461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:15.568143 28461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:08:15.568262 28461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 02:08:15.603296 28461 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:45909
I20260926 02:08:15.603344 28587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:45909 every 8 connection(s)
I20260926 02:08:15.604902 28461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:08:15.608590 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 28461
I20260926 02:08:15.608805 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:08:15.609437 19070 external_mini_cluster.cc:1428] Setting key e522e17f9ba9db030885142e3b6acfa0
I20260926 02:08:15.612519 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:46255
--builtin_ntp_servers=127.18.159.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 02:08:15.619349 28590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:15.607816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:33853 (local address 127.18.159.190:46255)
0926 02:08:15.608322 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:15.608328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:15.609753 (+ 1425us) server_negotiation.cc:407] Connection header received
0926 02:08:15.611987 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:15.611993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:15.612079 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:15.612233 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:15.613376 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:15.614603 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:15.616134 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:15.616896 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:15.617041 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:15.618456 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:15.618465 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:15.618473 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:08:15.618501 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:15.618575 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:08:15.618594 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:15.618693 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:15.618958 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 02:08:15.619089 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":273,"threads_started":1}
I20260926 02:08:15.620954 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:15.608462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:46255 (local address 127.18.159.129:33853)
0926 02:08:15.609560 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:15.609607 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:15.611713 (+ 2106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:15.612447 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:15.612456 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:15.612558 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:15.613203 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:15.613220 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:15.614747 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:15.614751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:15.615989 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:15.616000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:15.617123 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:15.617128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:15.617379 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:15.618121 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:08:15.618124 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:15.618170 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:08:15.618255 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:08:15.618708 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:08:15.618716 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:08:15.618722 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 02:08:15.618913 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 02:08:15.619427 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":346,"threads_started":1}
I20260926 02:08:15.622615 28588 heartbeater.cc:347] Connected to a master server at 127.18.159.190:46255
I20260926 02:08:15.622946 28588 heartbeater.cc:464] Registering TS with master...
I20260926 02:08:15.623756 28588 heartbeater.cc:511] Master 127.18.159.190:46255 requested a full tablet report, sending...
I20260926 02:08:15.625986 28402 ts_manager.cc:194] Registered new tserver with Master: 7b48fb314eb5449f90c46827f3bf8889 (127.18.159.129:45909)
I20260926 02:08:15.627525 28402 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.159.129:33853
W20260926 02:08:15.955749 28592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:08:15.956130 28592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:08:15.956313 28592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:08:15.963927 28592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:08:15.964105 28592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:08:15.964306 28592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:08:15.973788 28592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:46255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:08:15.975157 28592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:08:15.977075 28592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:08:15.991088 28597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:15.991225 28598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:15.991247 28600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:08:15.992214 28592 server_base.cc:1034] running on GCE node
I20260926 02:08:15.992897 28592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:08:15.994081 28592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:08:15.995301 28592 hybrid_clock.cc:648] HybridClock initialized: now 1790388495995223 us; error 75 us; skew 500 ppm
I20260926 02:08:15.995615 28592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:08:15.998202 28592 webserver.cc:460] Webserver started at http://127.18.159.130:33785/ using document root <none> and password file <none>
I20260926 02:08:15.998937 28592 fs_manager.cc:359] Metadata directory not provided
I20260926 02:08:15.999037 28592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:08:15.999315 28592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:08:16.002389 28592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "692463dbbbaf4368a23a0ac443c47c74"
format_stamp: "Formatted at 2026-09-26 02:08:15 on dist-test-slave-03rn"
server_key: "79cc4cf1d8850451d783362b19a5f73f"
server_key_iv: "5c1572dfe2b032681b319baca50c706c"
server_key_version: "encryptionkey@0"
I20260926 02:08:16.003260 28592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "692463dbbbaf4368a23a0ac443c47c74"
format_stamp: "Formatted at 2026-09-26 02:08:15 on dist-test-slave-03rn"
server_key: "79cc4cf1d8850451d783362b19a5f73f"
server_key_iv: "5c1572dfe2b032681b319baca50c706c"
server_key_version: "encryptionkey@0"
I20260926 02:08:16.008882 28592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260926 02:08:16.012642 28606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:16.014108 28592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:08:16.014315 28592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "692463dbbbaf4368a23a0ac443c47c74"
format_stamp: "Formatted at 2026-09-26 02:08:15 on dist-test-slave-03rn"
server_key: "79cc4cf1d8850451d783362b19a5f73f"
server_key_iv: "5c1572dfe2b032681b319baca50c706c"
server_key_version: "encryptionkey@0"
I20260926 02:08:16.014554 28592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:08:16.046769 28592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:08:16.047791 28592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:08:16.048091 28592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:08:16.049103 28592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:08:16.050521 28592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:08:16.050606 28592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:16.050680 28592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:08:16.050737 28592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:16.085095 28592 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:36599
I20260926 02:08:16.085178 28718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:36599 every 8 connection(s)
I20260926 02:08:16.086453 28592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:08:16.088987 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 28592
I20260926 02:08:16.089182 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:08:16.089726 19070 external_mini_cluster.cc:1428] Setting key 53e666dbf2af2e7bfda91c01338fdd15
I20260926 02:08:16.092988 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:46255
--builtin_ntp_servers=127.18.159.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 02:08:16.101912 28590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:16.089281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:46697 (local address 127.18.159.190:46255)
0926 02:08:16.089511 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:16.089518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:16.091195 (+ 1677us) server_negotiation.cc:407] Connection header received
0926 02:08:16.093762 (+ 2567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:16.093766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:16.093833 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:16.093912 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:16.095439 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:16.096131 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:16.098678 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:16.099093 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:16.099192 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:16.100886 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:16.100895 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:16.100901 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:08:16.100927 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:16.100970 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:08:16.100988 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:16.101092 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:16.101368 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 02:08:16.101485 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:08:16.102855 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:16.089918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:46255 (local address 127.18.159.130:46697)
0926 02:08:16.091040 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:16.091078 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:16.093417 (+ 2339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:16.094215 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:16.094231 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:16.094382 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:16.095260 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:16.095287 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:16.096338 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:16.096346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:16.098521 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:16.098539 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:16.099324 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:16.099332 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:16.099683 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:16.100554 (+ 871us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:08:16.100556 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:16.100598 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:08:16.100679 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:08:16.101121 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:08:16.101131 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:08:16.101138 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 02:08:16.101320 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 02:08:16.101664 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":360,"threads_started":1}
I20260926 02:08:16.104516 28719 heartbeater.cc:347] Connected to a master server at 127.18.159.190:46255
I20260926 02:08:16.104833 28719 heartbeater.cc:464] Registering TS with master...
I20260926 02:08:16.105726 28719 heartbeater.cc:511] Master 127.18.159.190:46255 requested a full tablet report, sending...
I20260926 02:08:16.107187 28402 ts_manager.cc:194] Registered new tserver with Master: 692463dbbbaf4368a23a0ac443c47c74 (127.18.159.130:36599)
I20260926 02:08:16.108042 28402 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.159.130:46697
W20260926 02:08:16.443881 28722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:08:16.444320 28722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:08:16.444490 28722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:08:16.452461 28722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:08:16.452580 28722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:08:16.452750 28722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:08:16.462502 28722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:46255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:08:16.463860 28722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:08:16.465780 28722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:08:16.479900 28727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:16.480039 28728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:16.480091 28730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:08:16.481024 28722 server_base.cc:1034] running on GCE node
I20260926 02:08:16.481956 28722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:08:16.483114 28722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:08:16.484285 28722 hybrid_clock.cc:648] HybridClock initialized: now 1790388496484234 us; error 43 us; skew 500 ppm
I20260926 02:08:16.484638 28722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:08:16.492797 28722 webserver.cc:460] Webserver started at http://127.18.159.131:33349/ using document root <none> and password file <none>
I20260926 02:08:16.493628 28722 fs_manager.cc:359] Metadata directory not provided
I20260926 02:08:16.493762 28722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:08:16.494124 28722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:08:16.497226 28722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "9dbe8f2ee63e43b69886bcd0c69f1c23"
format_stamp: "Formatted at 2026-09-26 02:08:16 on dist-test-slave-03rn"
server_key: "3439456e2bc8564b84e84331a339fdb4"
server_key_iv: "7d4f765b025fd7c9c9c8b6c02d4dc321"
server_key_version: "encryptionkey@0"
I20260926 02:08:16.498054 28722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "9dbe8f2ee63e43b69886bcd0c69f1c23"
format_stamp: "Formatted at 2026-09-26 02:08:16 on dist-test-slave-03rn"
server_key: "3439456e2bc8564b84e84331a339fdb4"
server_key_iv: "7d4f765b025fd7c9c9c8b6c02d4dc321"
server_key_version: "encryptionkey@0"
I20260926 02:08:16.503652 28722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:08:16.507615 28736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:16.509204 28722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:08:16.509403 28722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "9dbe8f2ee63e43b69886bcd0c69f1c23"
format_stamp: "Formatted at 2026-09-26 02:08:16 on dist-test-slave-03rn"
server_key: "3439456e2bc8564b84e84331a339fdb4"
server_key_iv: "7d4f765b025fd7c9c9c8b6c02d4dc321"
server_key_version: "encryptionkey@0"
I20260926 02:08:16.509641 28722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:08:16.525350 28722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:08:16.526297 28722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:08:16.526619 28722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:08:16.527680 28722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:08:16.529500 28722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:08:16.529600 28722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:16.529765 28722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:08:16.529848 28722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:16.564517 28722 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:46309
I20260926 02:08:16.564612 28848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:46309 every 8 connection(s)
I20260926 02:08:16.565891 28722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:08:16.569802 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 28722
I20260926 02:08:16.570048 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:08:16.570513 19070 external_mini_cluster.cc:1428] Setting key 1e136f4401e27c61aec2691b8913d79e
I20260926 02:08:16.579746 28590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:16.568830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:35907 (local address 127.18.159.190:46255)
0926 02:08:16.569032 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:16.569037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:16.570871 (+ 1834us) server_negotiation.cc:407] Connection header received
0926 02:08:16.572967 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:16.572972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:16.573064 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:16.573182 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:16.574426 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:16.575161 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:16.576902 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:16.577336 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:16.577477 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:16.578902 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:16.578907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:16.578911 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:08:16.578930 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:16.578961 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:08:16.578973 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:16.579072 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:16.579301 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 02:08:16.579402 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:08:16.580859 28851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:16.569464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:46255 (local address 127.18.159.131:35907)
0926 02:08:16.570657 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:16.570712 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:16.572683 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:16.573479 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:16.573493 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:16.573629 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:16.574256 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:16.574273 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:16.575327 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:16.575332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:16.576754 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:16.576766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:16.577569 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:16.577577 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:16.577815 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:16.578544 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:08:16.578547 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:16.578592 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:08:16.578673 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:08:16.579079 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:08:16.579092 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:08:16.579102 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 02:08:16.579293 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 02:08:16.579710 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":376,"threads_started":1}
I20260926 02:08:16.582345 28849 heartbeater.cc:347] Connected to a master server at 127.18.159.190:46255
I20260926 02:08:16.582685 28849 heartbeater.cc:464] Registering TS with master...
I20260926 02:08:16.583531 28849 heartbeater.cc:511] Master 127.18.159.190:46255 requested a full tablet report, sending...
I20260926 02:08:16.585249 28401 ts_manager.cc:194] Registered new tserver with Master: 9dbe8f2ee63e43b69886bcd0c69f1c23 (127.18.159.131:46309)
I20260926 02:08:16.586251 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:08:16.586371 28401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.159.131:35907
W20260926 02:08:16.593349 28590 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:16.591673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47042 (local address 127.18.159.190:46255)
0926 02:08:16.591847 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:16.591851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:16.592433 (+ 582us) server_negotiation.cc:407] Connection header received
0926 02:08:16.592578 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:16.592582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:16.592657 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:16.592749 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:16.593085 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47042: BlockingRecv error: recv got EOF from 127.0.0.1:47042 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:08:16.594247 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 28461
I20260926 02:08:16.603251 28858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:16.595121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48472 (local address 127.18.159.129:45909)
0926 02:08:16.595642 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:16.595649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:16.595674 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:08:16.595728 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:16.595734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:16.595954 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:16.596089 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:16.596506 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:16.597639 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:16.599660 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:16.600595 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:16.600730 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:16.601796 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:16.601804 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:16.601819 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:08:16.602442 (+ 623us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:16.602492 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:08:16.602515 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:16.602604 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:16.602805 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 02:08:16.602928 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":267,"threads_started":1}
I20260926 02:08:16.604626 28583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:08:16.689420 28588 heartbeater.cc:503] Master 127.18.159.190:46255 was elected leader, sending a full tablet report...
I20260926 02:08:16.691214 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 28461
I20260926 02:08:16.704957 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 28592
I20260926 02:08:16.713666 28860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:16.705960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.18.159.130:36599)
0926 02:08:16.706485 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:16.706492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:16.706519 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:08:16.706590 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:16.706597 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:16.706832 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:16.707042 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:16.707522 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:16.708667 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:16.710008 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:16.710604 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:16.710752 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:16.711972 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:16.711980 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:16.711995 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:08:16.712676 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:16.712726 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:08:16.712758 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:16.712906 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:16.713067 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 02:08:16.713207 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":271,"threads_started":1}
I20260926 02:08:16.715054 28714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:08:16.797236 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 28592
I20260926 02:08:16.813540 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 28722
I20260926 02:08:16.821961 28862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:16.814561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.18.159.131:46309)
0926 02:08:16.815352 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:16.815359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:16.815390 (+ 31us) server_negotiation.cc:407] Connection header received
0926 02:08:16.815449 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:16.815455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:16.815663 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:16.815809 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:16.816272 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:16.817327 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:16.818607 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:16.819234 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:16.819345 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:16.820445 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:16.820454 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:16.820468 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:08:16.821083 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:16.821141 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:08:16.821161 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:16.821255 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:16.821413 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 02:08:16.821569 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":497,"threads_started":1}
I20260926 02:08:16.823369 28844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:08:16.906775 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 28722
I20260926 02:08:16.923471 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 28371
I20260926 02:08:16.924727 28432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:08:17.027097 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 28371
2026-09-26T02:08:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2529 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T02:08:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:08:17Z Disabled control of system clock
I20260926 02:08:17.101609 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:46541
--webserver_interface=127.18.159.190
--webserver_port=0
--builtin_ntp_servers=127.18.159.148:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:46541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 02:08:17.445039 28868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:08:17.445483 28868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:08:17.445645 28868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:08:17.453087 28868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:08:17.453280 28868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:08:17.453385 28868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:08:17.453493 28868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:08:17.453583 28868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:08:17.463094 28868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:46541
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:46541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:08:17.464519 28868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:08:17.466405 28868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:08:17.478189 28874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:17.478163 28873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:17.478997 28876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:08:17.479888 28868 server_base.cc:1034] running on GCE node
I20260926 02:08:17.480809 28868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:08:17.482378 28868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:08:17.483620 28868 hybrid_clock.cc:648] HybridClock initialized: now 1790388497483549 us; error 74 us; skew 500 ppm
I20260926 02:08:17.484020 28868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:08:17.486598 28868 webserver.cc:460] Webserver started at http://127.18.159.190:38719/ using document root <none> and password file <none>
I20260926 02:08:17.487344 28868 fs_manager.cc:359] Metadata directory not provided
I20260926 02:08:17.487473 28868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:08:17.487799 28868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:08:17.490828 28868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "4803772fec3f4e55a6a0a9ae041fc185"
format_stamp: "Formatted at 2026-09-26 02:08:17 on dist-test-slave-03rn"
server_key: "cc87596316063b1904498c2c76ce8ad4"
server_key_iv: "a7e9cb96cbf33b4cf776ae0663bff7c3"
server_key_version: "encryptionkey@0"
I20260926 02:08:17.491703 28868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "4803772fec3f4e55a6a0a9ae041fc185"
format_stamp: "Formatted at 2026-09-26 02:08:17 on dist-test-slave-03rn"
server_key: "cc87596316063b1904498c2c76ce8ad4"
server_key_iv: "a7e9cb96cbf33b4cf776ae0663bff7c3"
server_key_version: "encryptionkey@0"
I20260926 02:08:17.497606 28868 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:08:17.501504 28882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:17.503149 28868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:08:17.503386 28868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "4803772fec3f4e55a6a0a9ae041fc185"
format_stamp: "Formatted at 2026-09-26 02:08:17 on dist-test-slave-03rn"
server_key: "cc87596316063b1904498c2c76ce8ad4"
server_key_iv: "a7e9cb96cbf33b4cf776ae0663bff7c3"
server_key_version: "encryptionkey@0"
I20260926 02:08:17.503651 28868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:08:17.510147 28868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:08:17.511031 28868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:08:17.511363 28868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:08:17.530606 28868 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:46541
I20260926 02:08:17.530645 28933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:46541 every 8 connection(s)
I20260926 02:08:17.532094 28868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:08:17.535691 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 28868
I20260926 02:08:17.535948 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:08:17.536319 28934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:08:17.536727 19070 external_mini_cluster.cc:1428] Setting key e6ad73493c2c11332e63a6065ce4a0fe
I20260926 02:08:17.546613 28937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:17.539091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.18.159.190:46541)
0926 02:08:17.540369 (+ 1278us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:17.540389 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:17.540440 (+ 51us) server_negotiation.cc:407] Connection header received
0926 02:08:17.542659 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:17.542688 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:17.543220 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:17.543618 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:17.544868 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:17.544874 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:17.544889 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:08:17.544904 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:17.544972 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:08:17.544988 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:17.545201 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:17.545312 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 02:08:17.545649 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":479,"threads_started":1}
I20260926 02:08:17.550153 28934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185: Bootstrap starting.
I20260926 02:08:17.553436 28934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185: Neither blocks nor log segments found. Creating new log.
I20260926 02:08:17.554675 28934 log.cc:824] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185: Log is configured to *not* fsync() on all Append() calls
I20260926 02:08:17.557847 28934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185: No bootstrap required, opened a new log
I20260926 02:08:17.565382 28934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4803772fec3f4e55a6a0a9ae041fc185" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 46541 } }
I20260926 02:08:17.565804 28934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:08:17.565933 28934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4803772fec3f4e55a6a0a9ae041fc185, State: Initialized, Role: FOLLOWER
I20260926 02:08:17.566553 28934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4803772fec3f4e55a6a0a9ae041fc185" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 46541 } }
I20260926 02:08:17.566757 28934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:08:17.566857 28934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:08:17.566993 28934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:08:17.569322 28934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4803772fec3f4e55a6a0a9ae041fc185" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 46541 } }
I20260926 02:08:17.569864 28934 leader_election.cc:302] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4803772fec3f4e55a6a0a9ae041fc185; no voters:
I20260926 02:08:17.570462 28934 leader_election.cc:288] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:08:17.570634 28939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:08:17.571943 28939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185 [term 1 LEADER]: Becoming Leader. State: Replica: 4803772fec3f4e55a6a0a9ae041fc185, State: Running, Role: LEADER
I20260926 02:08:17.572589 28939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4803772fec3f4e55a6a0a9ae041fc185" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 46541 } }
I20260926 02:08:17.572811 28934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:08:17.575800 28941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4803772fec3f4e55a6a0a9ae041fc185. Latest consensus state: current_term: 1 leader_uuid: "4803772fec3f4e55a6a0a9ae041fc185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4803772fec3f4e55a6a0a9ae041fc185" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 46541 } } }
I20260926 02:08:17.575870 28940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4803772fec3f4e55a6a0a9ae041fc185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4803772fec3f4e55a6a0a9ae041fc185" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 46541 } } }
I20260926 02:08:17.576327 28941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185 [sys.catalog]: This master's current role is: LEADER
I20260926 02:08:17.576332 28940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185 [sys.catalog]: This master's current role is: LEADER
I20260926 02:08:17.579111 28948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:08:17.584882 28948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:08:17.595157 28948 catalog_manager.cc:1347] Generated new cluster ID: 5bbd372bf0484c79bf1c4165230379d4
I20260926 02:08:17.595293 28948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:08:17.612002 28948 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:08:17.612586 28948 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:08:17.623170 28948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4803772fec3f4e55a6a0a9ae041fc185: Generated new TSK 0
I20260926 02:08:17.624271 28948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:08:17.633037 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:46541
--builtin_ntp_servers=127.18.159.148:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 02:08:18.021996 28958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:08:18.022359 28958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:08:18.022526 28958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:08:18.029973 28958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:08:18.030156 28958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:08:18.030355 28958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:08:18.040038 28958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:46541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:08:18.041455 28958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:08:18.043318 28958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:08:18.056999 28963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:18.056969 28966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:18.057137 28964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:08:18.057941 28958 server_base.cc:1034] running on GCE node
I20260926 02:08:18.058636 28958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:08:18.059652 28958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:08:18.060885 28958 hybrid_clock.cc:648] HybridClock initialized: now 1790388498060841 us; error 42 us; skew 500 ppm
I20260926 02:08:18.061275 28958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:08:18.063640 28958 webserver.cc:460] Webserver started at http://127.18.159.129:41665/ using document root <none> and password file <none>
I20260926 02:08:18.064483 28958 fs_manager.cc:359] Metadata directory not provided
I20260926 02:08:18.064630 28958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:08:18.064972 28958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:08:18.067981 28958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "1f34693560584422b8ebcf92344e32ff"
format_stamp: "Formatted at 2026-09-26 02:08:18 on dist-test-slave-03rn"
server_key: "429bc5b7a349d7bdf80bc1504d62b669"
server_key_iv: "08911973a48efa9ef33d874adc732361"
server_key_version: "encryptionkey@0"
I20260926 02:08:18.068962 28958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "1f34693560584422b8ebcf92344e32ff"
format_stamp: "Formatted at 2026-09-26 02:08:18 on dist-test-slave-03rn"
server_key: "429bc5b7a349d7bdf80bc1504d62b669"
server_key_iv: "08911973a48efa9ef33d874adc732361"
server_key_version: "encryptionkey@0"
I20260926 02:08:18.075176 28958 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:08:18.079339 28972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:18.080977 28958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:08:18.081211 28958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "1f34693560584422b8ebcf92344e32ff"
format_stamp: "Formatted at 2026-09-26 02:08:18 on dist-test-slave-03rn"
server_key: "429bc5b7a349d7bdf80bc1504d62b669"
server_key_iv: "08911973a48efa9ef33d874adc732361"
server_key_version: "encryptionkey@0"
I20260926 02:08:18.081466 28958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:08:18.088044 28958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:08:18.089011 28958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:08:18.089282 28958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:08:18.090214 28958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:08:18.091694 28958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:08:18.091784 28958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:18.091866 28958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:08:18.091921 28958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:18.126360 28958 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:45719
I20260926 02:08:18.126435 29084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:45719 every 8 connection(s)
I20260926 02:08:18.127734 28958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:08:18.135637 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 28958
I20260926 02:08:18.135878 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:08:18.136482 29087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:18.130438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:54387 (local address 127.18.159.190:46541)
0926 02:08:18.130882 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:18.130886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:18.132125 (+ 1239us) server_negotiation.cc:407] Connection header received
0926 02:08:18.133857 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:18.133861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:18.133913 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:18.134015 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:18.135499 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:18.135506 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:18.135512 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:08:18.135541 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:18.135596 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:08:18.135618 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:18.135765 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:18.136018 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 02:08:18.136088 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":244,"threads_started":1}
I20260926 02:08:18.136724 19070 external_mini_cluster.cc:1428] Setting key 68b1ef9d8963fd97d221eb7a67489c43
I20260926 02:08:18.137530 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:18.131076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:46541 (local address 127.18.159.129:54387)
0926 02:08:18.131955 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:18.131996 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:18.133569 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:18.134245 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:18.134253 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:18.134353 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:18.135150 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:08:18.135152 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:18.135192 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:08:18.135282 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:08:18.135746 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:08:18.135759 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:08:18.135768 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 02:08:18.136101 (+ 333us) client_negotiation.cc:240] Negotiation successful
0926 02:08:18.136379 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":259,"threads_started":1}
I20260926 02:08:18.139282 29085 heartbeater.cc:347] Connected to a master server at 127.18.159.190:46541
I20260926 02:08:18.139595 29085 heartbeater.cc:464] Registering TS with master...
I20260926 02:08:18.139743 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:46541
--builtin_ntp_servers=127.18.159.148:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 02:08:18.140316 29085 heartbeater.cc:511] Master 127.18.159.190:46541 requested a full tablet report, sending...
I20260926 02:08:18.142375 28898 ts_manager.cc:194] Registered new tserver with Master: 1f34693560584422b8ebcf92344e32ff (127.18.159.129:45719)
W20260926 02:08:18.489408 29089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:08:18.489710 29089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:08:18.489876 29089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:08:18.497320 29089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:08:18.497437 29089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:08:18.497633 29089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:08:18.507195 29089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:46541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:08:18.508558 29089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:08:18.510336 29089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:08:18.525100 29095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:18.525149 29097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:18.525714 29094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:08:18.526667 29089 server_base.cc:1034] running on GCE node
I20260926 02:08:18.527431 29089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:08:18.528554 29089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:08:18.529824 29089 hybrid_clock.cc:648] HybridClock initialized: now 1790388498529773 us; error 34 us; skew 500 ppm
I20260926 02:08:18.530236 29089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:08:18.533254 29089 webserver.cc:460] Webserver started at http://127.18.159.130:32801/ using document root <none> and password file <none>
I20260926 02:08:18.534157 29089 fs_manager.cc:359] Metadata directory not provided
I20260926 02:08:18.534281 29089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:08:18.534623 29089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:08:18.538566 29089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "4c95f7236b214d048711e94bb3e50d46"
format_stamp: "Formatted at 2026-09-26 02:08:18 on dist-test-slave-03rn"
server_key: "e718104a9df4c6a6bb49d681af2f2175"
server_key_iv: "8704d8b0b399ae107dac31084b8acdb5"
server_key_version: "encryptionkey@0"
I20260926 02:08:18.539683 29089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "4c95f7236b214d048711e94bb3e50d46"
format_stamp: "Formatted at 2026-09-26 02:08:18 on dist-test-slave-03rn"
server_key: "e718104a9df4c6a6bb49d681af2f2175"
server_key_iv: "8704d8b0b399ae107dac31084b8acdb5"
server_key_version: "encryptionkey@0"
I20260926 02:08:18.546576 29089 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260926 02:08:18.551143 29103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:18.552815 29089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:08:18.552964 29089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "4c95f7236b214d048711e94bb3e50d46"
format_stamp: "Formatted at 2026-09-26 02:08:18 on dist-test-slave-03rn"
server_key: "e718104a9df4c6a6bb49d681af2f2175"
server_key_iv: "8704d8b0b399ae107dac31084b8acdb5"
server_key_version: "encryptionkey@0"
I20260926 02:08:18.553154 29089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:08:18.558897 29089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:08:18.559703 29089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:08:18.559931 29089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:08:18.560973 29089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:08:18.562413 29089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:08:18.562512 29089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:18.562593 29089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:08:18.562654 29089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:18.596396 29089 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:38523
I20260926 02:08:18.596444 29215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:38523 every 8 connection(s)
I20260926 02:08:18.597815 29089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:08:18.603180 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 29089
I20260926 02:08:18.603406 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:08:18.603856 19070 external_mini_cluster.cc:1428] Setting key cd323a60b7deec8c9163fcab85050b5f
I20260926 02:08:18.607798 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:46541
--builtin_ntp_servers=127.18.159.148:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 02:08:18.608548 29087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:18.600776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:35887 (local address 127.18.159.190:46541)
0926 02:08:18.600955 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:18.600989 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:18.602626 (+ 1637us) server_negotiation.cc:407] Connection header received
0926 02:08:18.604756 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:18.604763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:18.604872 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:18.605030 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:18.607187 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:18.607194 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:18.607199 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:08:18.607222 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:18.607320 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:08:18.607388 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:18.607603 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:18.607901 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 02:08:18.608093 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:08:18.609462 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:18.601399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:46541 (local address 127.18.159.130:35887)
0926 02:08:18.602442 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:18.602488 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:18.604292 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:18.605340 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:18.605352 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:18.605500 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:18.606702 (+ 1202us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:08:18.606706 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:18.606782 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:08:18.606920 (+ 138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:08:18.607530 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:08:18.607544 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:08:18.607578 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 02:08:18.607947 (+ 369us) client_negotiation.cc:240] Negotiation successful
0926 02:08:18.608307 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":296,"threads_started":1}
I20260926 02:08:18.611150 29216 heartbeater.cc:347] Connected to a master server at 127.18.159.190:46541
I20260926 02:08:18.611522 29216 heartbeater.cc:464] Registering TS with master...
I20260926 02:08:18.612262 29216 heartbeater.cc:511] Master 127.18.159.190:46541 requested a full tablet report, sending...
I20260926 02:08:18.613965 28898 ts_manager.cc:194] Registered new tserver with Master: 4c95f7236b214d048711e94bb3e50d46 (127.18.159.130:38523)
W20260926 02:08:18.962198 29219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:08:18.962563 29219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:08:18.962735 29219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:08:18.970538 29219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:08:18.970722 29219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:08:18.970907 29219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:08:18.982481 29219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:46541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:08:18.984328 29219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:08:18.986677 29219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:08:19.000895 29225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:19.001067 29224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:19.001042 29227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:08:19.002226 29219 server_base.cc:1034] running on GCE node
I20260926 02:08:19.003046 29219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:08:19.004276 29219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:08:19.005587 29219 hybrid_clock.cc:648] HybridClock initialized: now 1790388499005547 us; error 63 us; skew 500 ppm
I20260926 02:08:19.005980 29219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:08:19.008741 29219 webserver.cc:460] Webserver started at http://127.18.159.131:38147/ using document root <none> and password file <none>
I20260926 02:08:19.009496 29219 fs_manager.cc:359] Metadata directory not provided
I20260926 02:08:19.009634 29219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:08:19.010004 29219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:08:19.013232 29219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "e0b6154787db4578b5e67da43c56f7c8"
format_stamp: "Formatted at 2026-09-26 02:08:19 on dist-test-slave-03rn"
server_key: "9fc66b6a3048c019615aae5d0e8b529e"
server_key_iv: "b12c20da860f828b3f4685a19bc3b063"
server_key_version: "encryptionkey@0"
I20260926 02:08:19.014099 29219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "e0b6154787db4578b5e67da43c56f7c8"
format_stamp: "Formatted at 2026-09-26 02:08:19 on dist-test-slave-03rn"
server_key: "9fc66b6a3048c019615aae5d0e8b529e"
server_key_iv: "b12c20da860f828b3f4685a19bc3b063"
server_key_version: "encryptionkey@0"
I20260926 02:08:19.019665 29219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:08:19.023427 29233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:19.025020 29219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:08:19.025219 29219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "e0b6154787db4578b5e67da43c56f7c8"
format_stamp: "Formatted at 2026-09-26 02:08:19 on dist-test-slave-03rn"
server_key: "9fc66b6a3048c019615aae5d0e8b529e"
server_key_iv: "b12c20da860f828b3f4685a19bc3b063"
server_key_version: "encryptionkey@0"
I20260926 02:08:19.025454 29219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:08:19.031327 29219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:08:19.032140 29219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:08:19.032501 29219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:08:19.033391 29219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:08:19.034718 29219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:08:19.034806 29219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:19.034888 29219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:08:19.034945 29219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:19.069335 29219 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:39483
I20260926 02:08:19.069396 29345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:39483 every 8 connection(s)
I20260926 02:08:19.070713 29219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:08:19.073424 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 29219
I20260926 02:08:19.073657 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:08:19.074178 19070 external_mini_cluster.cc:1428] Setting key b5ec41401a62ea334b70847724a178b4
I20260926 02:08:19.080241 29087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:19.073530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:52045 (local address 127.18.159.190:46541)
0926 02:08:19.073739 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:19.073745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:19.075925 (+ 2180us) server_negotiation.cc:407] Connection header received
0926 02:08:19.077934 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:19.077939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:19.077992 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:19.078074 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:19.079518 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:19.079522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:19.079526 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:08:19.079541 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:19.079582 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:08:19.079594 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:19.079689 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:19.079874 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 02:08:19.079937 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:08:19.081281 29348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:19.074249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:46541 (local address 127.18.159.131:52045)
0926 02:08:19.075763 (+ 1514us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:19.075816 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:19.077656 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:19.078293 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:19.078302 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:19.078424 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:19.079167 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:08:19.079169 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:19.079215 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:08:19.079308 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:08:19.079689 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:08:19.079696 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:08:19.079703 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 02:08:19.079936 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 02:08:19.080220 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":367,"threads_started":1}
I20260926 02:08:19.082731 29346 heartbeater.cc:347] Connected to a master server at 127.18.159.190:46541
I20260926 02:08:19.083022 29346 heartbeater.cc:464] Registering TS with master...
I20260926 02:08:19.083612 29346 heartbeater.cc:511] Master 127.18.159.190:46541 requested a full tablet report, sending...
I20260926 02:08:19.085202 28898 ts_manager.cc:194] Registered new tserver with Master: e0b6154787db4578b5e67da43c56f7c8 (127.18.159.131:39483)
I20260926 02:08:19.090462 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 02:08:19.097112 29087 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:08:19.095398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43736 (local address 127.18.159.190:46541)
0926 02:08:19.095604 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:19.095612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:19.096022 (+ 410us) server_negotiation.cc:407] Connection header received
0926 02:08:19.096278 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:19.096282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:19.096338 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:19.096474 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:19.096801 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43736: BlockingRecv error: recv got EOF from 127.0.0.1:43736 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:08:19.097838 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 28958
I20260926 02:08:19.101754 29355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:19.098869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40374 (local address 127.18.159.129:45719)
0926 02:08:19.099448 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:19.099455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:19.099481 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:08:19.099537 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:19.099542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:19.099824 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:19.099968 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:19.101028 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:19.101035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:19.101048 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:08:19.101068 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:19.101114 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:08:19.101128 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:19.101236 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:19.101314 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 02:08:19.101393 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":309,"threads_started":1}
I20260926 02:08:19.102974 29080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:08:19.183815 29085 heartbeater.cc:503] Master 127.18.159.190:46541 was elected leader, sending a full tablet report...
I20260926 02:08:19.186221 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 28958
I20260926 02:08:19.203646 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 29089
I20260926 02:08:19.207317 29357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:19.204747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.18.159.130:38523)
0926 02:08:19.205317 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:19.205324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:19.205352 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:08:19.205415 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:19.205428 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:19.205679 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:19.205808 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:19.206729 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:19.206736 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:19.206748 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:08:19.206765 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:19.206803 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:08:19.206816 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:19.206920 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:19.207006 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 02:08:19.207083 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":297,"threads_started":1}
I20260926 02:08:19.208859 29211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:08:19.290576 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 29089
I20260926 02:08:19.306941 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 29219
I20260926 02:08:19.311266 29359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:19.308125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33370 (local address 127.18.159.131:39483)
0926 02:08:19.308859 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:19.308866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:19.308898 (+ 32us) server_negotiation.cc:407] Connection header received
0926 02:08:19.308959 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:19.308966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:19.309195 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:19.309391 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:19.310549 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:19.310555 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:19.310567 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:08:19.310582 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:19.310620 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:08:19.310633 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:19.310814 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:19.310885 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 02:08:19.310983 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":376,"threads_started":1}
I20260926 02:08:19.312531 29341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:08:19.396284 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 29219
I20260926 02:08:19.412959 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 28868
I20260926 02:08:19.414202 28929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:08:19.516724 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 28868
2026-09-26T02:08:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2490 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:08:19 dist-test-slave-03rn krb5kdc[29364](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:08:19 dist-test-slave-03rn krb5kdc[29364](info): set up 2 sockets
Sep 26 02:08:19 dist-test-slave-03rn krb5kdc[29364](info): commencing operation
krb5kdc: starting...
W20260926 02:08:20.645485 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:08:20 dist-test-slave-03rn krb5kdc[29364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:08:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:08:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:08:20.897827 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:38437
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:38437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:08:21.246654 29377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:08:21.246985 29377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:08:21.247154 29377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:08:21.254959 29377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:08:21.255107 29377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:08:21.255208 29377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:08:21.255313 29377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:08:21.255409 29377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:08:21.266098 29377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:38437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:38437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:08:21.267560 29377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:08:21.269023 29377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:08:21.282115 29382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:21.282351 29383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:21.282179 29385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:08:21.283584 29377 server_base.cc:1034] running on GCE node
I20260926 02:08:21.284677 29377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:08:21.286448 29377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:08:21.287743 29377 hybrid_clock.cc:648] HybridClock initialized: now 1790388501287701 us; error 80 us; skew 500 ppm
I20260926 02:08:21.288257 29377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:08:21 dist-test-slave-03rn krb5kdc[29364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388501, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:08:21.291801 29377 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:08:21.293684 29377 webserver.cc:460] Webserver started at http://127.18.159.190:35939/ using document root <none> and password file <none>
I20260926 02:08:21.294456 29377 fs_manager.cc:359] Metadata directory not provided
I20260926 02:08:21.294623 29377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:08:21.294948 29377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:08:21.298148 29377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "513ecca5256248f9ba4336608e4aaef5"
format_stamp: "Formatted at 2026-09-26 02:08:21 on dist-test-slave-03rn"
server_key: "c3e5a5971879ec6625c99316b70ef5e9"
server_key_iv: "3dfaf4474a21434929d41f4b937ea358"
server_key_version: "encryptionkey@0"
I20260926 02:08:21.299031 29377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "513ecca5256248f9ba4336608e4aaef5"
format_stamp: "Formatted at 2026-09-26 02:08:21 on dist-test-slave-03rn"
server_key: "c3e5a5971879ec6625c99316b70ef5e9"
server_key_iv: "3dfaf4474a21434929d41f4b937ea358"
server_key_version: "encryptionkey@0"
I20260926 02:08:21.304773 29377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:08:21.308609 29392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:21.310096 29377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:08:21.310323 29377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "513ecca5256248f9ba4336608e4aaef5"
format_stamp: "Formatted at 2026-09-26 02:08:21 on dist-test-slave-03rn"
server_key: "c3e5a5971879ec6625c99316b70ef5e9"
server_key_iv: "3dfaf4474a21434929d41f4b937ea358"
server_key_version: "encryptionkey@0"
I20260926 02:08:21.310544 29377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:08:21.348789 29377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:08:21.353363 29377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:08:21.353739 29377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:08:21.373653 29377 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:38437
I20260926 02:08:21.373692 29443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:38437 every 8 connection(s)
I20260926 02:08:21.375108 29377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:08:21.379113 29444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:08:21.383569 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 29377
I20260926 02:08:21.383826 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:08:21.384680 19070 external_mini_cluster.cc:1428] Setting key e9cf8fbd3253c64c0fe3b93c9d24dfc3
I20260926 02:08:21.391466 29444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5: Bootstrap starting.
I20260926 02:08:21.394757 29444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5: Neither blocks nor log segments found. Creating new log.
I20260926 02:08:21.395860 29444 log.cc:824] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5: Log is configured to *not* fsync() on all Append() calls
I20260926 02:08:21.399228 29444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5: No bootstrap required, opened a new log
Sep 26 02:08:21 dist-test-slave-03rn krb5kdc[29364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:08:21.409097 29444 raft_consensus.cc:348] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513ecca5256248f9ba4336608e4aaef5" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38437 } }
I20260926 02:08:21.409493 29444 raft_consensus.cc:374] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:08:21.409632 29444 raft_consensus.cc:729] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513ecca5256248f9ba4336608e4aaef5, State: Initialized, Role: FOLLOWER
I20260926 02:08:21.409832 29447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:21.387052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35762 (local address 127.18.159.190:38437)
0926 02:08:21.387873 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:21.387886 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:21.387964 (+ 78us) server_negotiation.cc:407] Connection header received
0926 02:08:21.390076 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:21.390102 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:21.390532 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:21.390954 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:21.392582 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:21.393944 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:21.395327 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:21.396128 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:21.396384 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:21.400916 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:21.400925 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:21.400943 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:08:21.400987 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:21.404568 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:08:21.405633 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:08:21.405645 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:08:21.405653 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:08:21.405853 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:08:21.406406 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:08:21.406410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:08:21.406412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:08:21.407085 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:21.407361 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:21.407912 (+ 551us) server_negotiation.cc:299] Negotiation successful
0926 02:08:21.408490 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":383,"threads_started":1}
I20260926 02:08:21.410387 29444 consensus_queue.cc:260] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513ecca5256248f9ba4336608e4aaef5" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38437 } }
I20260926 02:08:21.410681 29444 raft_consensus.cc:388] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:08:21.410826 29444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:08:21.411007 29444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:08:21.414019 29444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513ecca5256248f9ba4336608e4aaef5" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38437 } }
I20260926 02:08:21.414618 29444 leader_election.cc:302] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 513ecca5256248f9ba4336608e4aaef5; no voters:
I20260926 02:08:21.415261 29444 leader_election.cc:288] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:08:21.415498 29449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:08:21.416343 29449 raft_consensus.cc:686] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5 [term 1 LEADER]: Becoming Leader. State: Replica: 513ecca5256248f9ba4336608e4aaef5, State: Running, Role: LEADER
I20260926 02:08:21.417438 29449 consensus_queue.cc:237] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513ecca5256248f9ba4336608e4aaef5" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38437 } }
I20260926 02:08:21.417804 29444 sys_catalog.cc:564] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:08:21.420919 29451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 513ecca5256248f9ba4336608e4aaef5. Latest consensus state: current_term: 1 leader_uuid: "513ecca5256248f9ba4336608e4aaef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513ecca5256248f9ba4336608e4aaef5" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38437 } } }
I20260926 02:08:21.420898 29450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "513ecca5256248f9ba4336608e4aaef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513ecca5256248f9ba4336608e4aaef5" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38437 } } }
I20260926 02:08:21.421327 29451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5 [sys.catalog]: This master's current role is: LEADER
I20260926 02:08:21.421394 29450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5 [sys.catalog]: This master's current role is: LEADER
I20260926 02:08:21.425217 29455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:08:21.431063 29455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:08:21.441349 29455 catalog_manager.cc:1347] Generated new cluster ID: c5276f0e8f454608ab4cf03db1da5e00
I20260926 02:08:21.441489 29455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:08:21.489462 29455 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:08:21.490867 29455 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:08:21.498281 29455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5: Generated new TSK 0
I20260926 02:08:21.499289 29455 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.159.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
I20260926 02:08:21.572880 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:38437
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:08:21.929661 29470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:08:21.929976 29470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:08:21.930080 29470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:08:21.937592 29470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:08:21.937722 29470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:08:21.937839 29470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:08:21.948573 29470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:38437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:08:21.949879 29470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:08:21.951153 29470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:08:21.965379 29475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:21.965417 29476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:21.965585 29478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:08:21.966449 29470 server_base.cc:1034] running on GCE node
I20260926 02:08:21.967239 29470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:08:21.968416 29470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:08:21.969661 29470 hybrid_clock.cc:648] HybridClock initialized: now 1790388501969607 us; error 65 us; skew 500 ppm
I20260926 02:08:21.970021 29470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:08:21 dist-test-slave-03rn krb5kdc[29364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388501, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:08:21.992878 29470 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:08:21.994539 29470 webserver.cc:460] Webserver started at http://127.18.159.129:40605/ using document root <none> and password file <none>
I20260926 02:08:21.995296 29470 fs_manager.cc:359] Metadata directory not provided
I20260926 02:08:21.995462 29470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:08:21.995779 29470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:08:21.999099 29470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "e694f452f67c4491b47761eb7c1d2ee5"
format_stamp: "Formatted at 2026-09-26 02:08:21 on dist-test-slave-03rn"
server_key: "477ac97d495242edfcdf5bee03adf80a"
server_key_iv: "6cb2a1a203947913cb995a6fb64e9bd7"
server_key_version: "encryptionkey@0"
I20260926 02:08:21.999931 29470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "e694f452f67c4491b47761eb7c1d2ee5"
format_stamp: "Formatted at 2026-09-26 02:08:21 on dist-test-slave-03rn"
server_key: "477ac97d495242edfcdf5bee03adf80a"
server_key_iv: "6cb2a1a203947913cb995a6fb64e9bd7"
server_key_version: "encryptionkey@0"
I20260926 02:08:22.006012 29470 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260926 02:08:22.010213 29485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:22.011710 29470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:08:22.011925 29470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "e694f452f67c4491b47761eb7c1d2ee5"
format_stamp: "Formatted at 2026-09-26 02:08:21 on dist-test-slave-03rn"
server_key: "477ac97d495242edfcdf5bee03adf80a"
server_key_iv: "6cb2a1a203947913cb995a6fb64e9bd7"
server_key_version: "encryptionkey@0"
I20260926 02:08:22.012156 29470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:08:22.036309 29470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:08:22.041059 29470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:08:22.041383 29470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:08:22.042414 29470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:08:22.043799 29470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:08:22.043916 29470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:22.044096 29470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:08:22.044207 29470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:22.078612 29470 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:39879
I20260926 02:08:22.078663 29597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:39879 every 8 connection(s)
I20260926 02:08:22.079972 29470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:08:22.088594 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 29470
I20260926 02:08:22.088840 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:08:22.089558 19070 external_mini_cluster.cc:1428] Setting key 6d50e357637868c7d6f571c42987d220
Sep 26 02:08:22 dist-test-slave-03rn krb5kdc[29364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388501, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:08:22.103294 29600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:22.082737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:39167 (local address 127.18.159.190:38437)
0926 02:08:22.083191 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:22.083195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:22.084302 (+ 1107us) server_negotiation.cc:407] Connection header received
0926 02:08:22.086242 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:22.086248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:22.086354 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:22.086497 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:22.088458 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:22.089377 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:22.091612 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:22.092272 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:22.092424 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:22.096139 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:22.096148 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:22.096183 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:08:22.096232 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:22.099525 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:08:22.100628 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:08:22.100632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:08:22.100635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:08:22.100771 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:08:22.101358 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:08:22.101362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:08:22.101365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:08:22.101695 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:22.101815 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:22.102827 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0926 02:08:22.103021 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":224,"threads_started":1}
I20260926 02:08:22.104884 29601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:22.083333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:38437 (local address 127.18.159.129:39167)
0926 02:08:22.084055 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:22.084096 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:22.085894 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:22.086787 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:22.086797 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:22.087377 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:22.088096 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:22.088190 (+ 94us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:22.089572 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:22.089581 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:22.091432 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:22.091454 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:22.092499 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:22.092505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:22.092726 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:22.093484 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:22.093508 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:22.095855 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:08:22.099704 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:08:22.099718 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:08:22.099742 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:08:22.100486 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:08:22.100889 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:08:22.100897 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:08:22.100903 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:08:22.101157 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:08:22.101845 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:08:22.101857 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:08:22.102348 (+ 491us) client_negotiation.cc:769] Sending connection context
0926 02:08:22.102694 (+ 346us) client_negotiation.cc:240] Negotiation successful
0926 02:08:22.103309 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":282,"threads_started":1}
I20260926 02:08:22.106617 29598 heartbeater.cc:347] Connected to a master server at 127.18.159.190:38437
I20260926 02:08:22.106950 29598 heartbeater.cc:464] Registering TS with master...
I20260926 02:08:22.107720 29598 heartbeater.cc:511] Master 127.18.159.190:38437 requested a full tablet report, sending...
I20260926 02:08:22.109932 29409 ts_manager.cc:194] Registered new tserver with Master: e694f452f67c4491b47761eb7c1d2ee5 (127.18.159.129:39879)
I20260926 02:08:22.111521 29409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:39167
WARNING: no policy specified for kudu/127.18.159.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
I20260926 02:08:22.147800 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:38437
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:08:22.491500 29604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:08:22.491811 29604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:08:22.491997 29604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:08:22.499951 29604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:08:22.500072 29604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:08:22.500319 29604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:08:22.510630 29604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:38437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:08:22.512017 29604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:08:22.513473 29604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:08:22.528071 29610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:22.527978 29612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:22.528074 29609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:08:22.529063 29604 server_base.cc:1034] running on GCE node
I20260926 02:08:22.529843 29604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:08:22.530972 29604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:08:22.532274 29604 hybrid_clock.cc:648] HybridClock initialized: now 1790388502532214 us; error 46 us; skew 500 ppm
I20260926 02:08:22.532641 29604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:08:22 dist-test-slave-03rn krb5kdc[29364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388502, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:08:22.535974 29604 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:08:22.537875 29604 webserver.cc:460] Webserver started at http://127.18.159.130:41465/ using document root <none> and password file <none>
I20260926 02:08:22.538628 29604 fs_manager.cc:359] Metadata directory not provided
I20260926 02:08:22.538791 29604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:08:22.539155 29604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:08:22.542307 29604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "040e77d1f4e84951b3cbd58c68662450"
format_stamp: "Formatted at 2026-09-26 02:08:22 on dist-test-slave-03rn"
server_key: "87d94f5bc5e22b1eb90f992a1cbdb113"
server_key_iv: "e20eb641f1f7ac290ce12f15e6ddfd24"
server_key_version: "encryptionkey@0"
I20260926 02:08:22.543177 29604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "040e77d1f4e84951b3cbd58c68662450"
format_stamp: "Formatted at 2026-09-26 02:08:22 on dist-test-slave-03rn"
server_key: "87d94f5bc5e22b1eb90f992a1cbdb113"
server_key_iv: "e20eb641f1f7ac290ce12f15e6ddfd24"
server_key_version: "encryptionkey@0"
I20260926 02:08:22.549031 29604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:08:22.552879 29619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:22.554416 29604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:08:22.554638 29604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "040e77d1f4e84951b3cbd58c68662450"
format_stamp: "Formatted at 2026-09-26 02:08:22 on dist-test-slave-03rn"
server_key: "87d94f5bc5e22b1eb90f992a1cbdb113"
server_key_iv: "e20eb641f1f7ac290ce12f15e6ddfd24"
server_key_version: "encryptionkey@0"
I20260926 02:08:22.554877 29604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:08:22.573932 29604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:08:22.578244 29604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:08:22.578588 29604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:08:22.579634 29604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:08:22.581161 29604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:08:22.581275 29604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:22.581440 29604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:08:22.581518 29604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:22.617352 29604 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:40563
I20260926 02:08:22.617395 29731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:40563 every 8 connection(s)
I20260926 02:08:22.618979 29604 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:08:22.622618 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 29604
I20260926 02:08:22.622860 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:08:22.623477 19070 external_mini_cluster.cc:1428] Setting key adf36571efc801349325b30036979b39
Sep 26 02:08:22 dist-test-slave-03rn krb5kdc[29364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388502, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.159.131@KRBTEST.COM; defaulting to no policy
I20260926 02:08:22.653775 29734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:22.622096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:38795 (local address 127.18.159.190:38437)
0926 02:08:22.625140 (+ 3044us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:22.625146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:22.633047 (+ 7901us) server_negotiation.cc:407] Connection header received
0926 02:08:22.635143 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:22.635149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:22.635252 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:22.635390 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:22.637856 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:22.638638 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:22.641406 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:22.641858 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:22.641968 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:22.646871 (+ 4903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:22.646878 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:22.646883 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:08:22.646925 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:22.649912 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:08:22.651113 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:08:22.651118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:08:22.651120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:08:22.651272 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:08:22.651896 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:08:22.651900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:08:22.651902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:08:22.652345 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:22.652462 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:22.653195 (+ 733us) server_negotiation.cc:299] Negotiation successful
0926 02:08:22.653427 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2907,"thread_start_us":2783,"threads_started":1}
I20260926 02:08:22.654772 29735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:22.622926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:38437 (local address 127.18.159.130:38795)
0926 02:08:22.632813 (+ 9887us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:22.632879 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:22.634793 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:22.635689 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:22.635700 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:22.636649 (+ 949us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:22.637600 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:22.637632 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:22.638843 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:22.638851 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:22.641200 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:22.641220 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:22.642142 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:22.642151 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:22.642504 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:22.643286 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:22.643327 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:22.646504 (+ 3177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:08:22.650143 (+ 3639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:08:22.650155 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:08:22.650178 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:08:22.650911 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:08:22.651415 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:08:22.651423 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:08:22.651430 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:08:22.651723 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:08:22.652490 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:08:22.652497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:08:22.652855 (+ 358us) client_negotiation.cc:769] Sending connection context
0926 02:08:22.653095 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 02:08:22.653594 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9584,"thread_start_us":9360,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.131@KRBTEST.COM" created.
I20260926 02:08:22.656564 29732 heartbeater.cc:347] Connected to a master server at 127.18.159.190:38437
I20260926 02:08:22.656920 29732 heartbeater.cc:464] Registering TS with master...
I20260926 02:08:22.657898 29732 heartbeater.cc:511] Master 127.18.159.190:38437 requested a full tablet report, sending...
I20260926 02:08:22.659780 29409 ts_manager.cc:194] Registered new tserver with Master: 040e77d1f4e84951b3cbd58c68662450 (127.18.159.130:40563)
I20260926 02:08:22.660993 29409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:38795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
I20260926 02:08:22.689395 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:38437
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:08:23.026736 29738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:08:23.027036 29738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:08:23.027217 29738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:08:23.034529 29738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:08:23.034655 29738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:08:23.034861 29738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:08:23.044718 29738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:38437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:08:23.046130 29738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:08:23.047530 29738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:08:23.061272 29744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:23.061311 29746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:08:23.061229 29743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:08:23.062605 29738 server_base.cc:1034] running on GCE node
I20260926 02:08:23.063294 29738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:08:23.064278 29738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:08:23.065485 29738 hybrid_clock.cc:648] HybridClock initialized: now 1790388503065406 us; error 74 us; skew 500 ppm
I20260926 02:08:23.065809 29738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:08:23 dist-test-slave-03rn krb5kdc[29364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388503, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:08:23.069088 29738 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:08:23.070643 29738 webserver.cc:460] Webserver started at http://127.18.159.131:38347/ using document root <none> and password file <none>
I20260926 02:08:23.071365 29738 fs_manager.cc:359] Metadata directory not provided
I20260926 02:08:23.071507 29738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:08:23.071830 29738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:08:23.074836 29738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "de4b59f2b8ed4485a3e9b6ba6dfc520b"
format_stamp: "Formatted at 2026-09-26 02:08:23 on dist-test-slave-03rn"
server_key: "30af531f9823e5b70cfce7c82a0f35b2"
server_key_iv: "85af5d256a2b8c0ef51bb73892bc2c2a"
server_key_version: "encryptionkey@0"
I20260926 02:08:23.075665 29738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "de4b59f2b8ed4485a3e9b6ba6dfc520b"
format_stamp: "Formatted at 2026-09-26 02:08:23 on dist-test-slave-03rn"
server_key: "30af531f9823e5b70cfce7c82a0f35b2"
server_key_iv: "85af5d256a2b8c0ef51bb73892bc2c2a"
server_key_version: "encryptionkey@0"
I20260926 02:08:23.081997 29738 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 02:08:23.086259 29753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:23.087790 29738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:08:23.088006 29738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "de4b59f2b8ed4485a3e9b6ba6dfc520b"
format_stamp: "Formatted at 2026-09-26 02:08:23 on dist-test-slave-03rn"
server_key: "30af531f9823e5b70cfce7c82a0f35b2"
server_key_iv: "85af5d256a2b8c0ef51bb73892bc2c2a"
server_key_version: "encryptionkey@0"
I20260926 02:08:23.088270 29738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:08:23.104971 29738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:08:23.109194 29738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:08:23.109521 29738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:08:23.110549 29738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:08:23.111927 29738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:08:23.112008 29738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:08:23.112141 29738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:08:23.112226 29738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 02:08:23.115095 29598 heartbeater.cc:503] Master 127.18.159.190:38437 was elected leader, sending a full tablet report...
I20260926 02:08:23.147328 29738 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:33337
I20260926 02:08:23.147365 29865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:33337 every 8 connection(s)
I20260926 02:08:23.148723 29738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:08:23.154382 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 29738
I20260926 02:08:23.154589 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:08:23.155118 19070 external_mini_cluster.cc:1428] Setting key 1a857935b209cf9d26d6cde200251f98
Sep 26 02:08:23 dist-test-slave-03rn krb5kdc[29364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388503, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:08:23.170678 29734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:23.151492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:51455 (local address 127.18.159.190:38437)
0926 02:08:23.151684 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:23.151689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:23.152911 (+ 1222us) server_negotiation.cc:407] Connection header received
0926 02:08:23.155213 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:23.155246 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:23.155337 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:23.155467 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:23.157765 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.158942 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.160830 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.161232 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.161331 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:23.164999 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:23.165005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:23.165008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:08:23.165041 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:23.167502 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:08:23.168269 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:08:23.168273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:08:23.168276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:08:23.168395 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:08:23.168794 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:08:23.168797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:08:23.168799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:08:23.169050 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:23.169228 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:23.170080 (+ 852us) server_negotiation.cc:299] Negotiation successful
0926 02:08:23.170275 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:08:23.171830 29868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:23.151917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:38437 (local address 127.18.159.131:51455)
0926 02:08:23.152731 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:23.152769 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:23.154827 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:23.155794 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:23.155807 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:23.156696 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:08:23.157532 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:23.157564 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.159079 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.159083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:23.160680 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:23.160693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.161468 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.161475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:23.161761 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:23.162419 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:08:23.162442 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:08:23.164762 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:08:23.167641 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:08:23.167648 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:08:23.167662 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:08:23.168096 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:08:23.168494 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:08:23.168497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:08:23.168499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:08:23.168686 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:08:23.169185 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:08:23.169197 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:08:23.169646 (+ 449us) client_negotiation.cc:769] Sending connection context
0926 02:08:23.169958 (+ 312us) client_negotiation.cc:240] Negotiation successful
0926 02:08:23.170548 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":378,"threads_started":1}
I20260926 02:08:23.173326 29866 heartbeater.cc:347] Connected to a master server at 127.18.159.190:38437
I20260926 02:08:23.173682 29866 heartbeater.cc:464] Registering TS with master...
I20260926 02:08:23.174454 29866 heartbeater.cc:511] Master 127.18.159.190:38437 requested a full tablet report, sending...
I20260926 02:08:23.176045 29409 ts_manager.cc:194] Registered new tserver with Master: de4b59f2b8ed4485a3e9b6ba6dfc520b (127.18.159.131:33337)
I20260926 02:08:23.176940 29409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:51455
I20260926 02:08:23.184602 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:08:23.202346 29734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:23.189542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35768 (local address 127.18.159.190:38437)
0926 02:08:23.189726 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:23.189731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:23.190328 (+ 597us) server_negotiation.cc:407] Connection header received
0926 02:08:23.190459 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:23.190463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:23.190536 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:23.190651 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:23.192279 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.193187 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.194424 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.194830 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.194924 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:23.197268 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:23.197272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:23.197276 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:08:23.197310 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:23.199944 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:08:23.200620 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:08:23.200624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:08:23.200626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:08:23.200754 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:08:23.201080 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:08:23.201084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:08:23.201086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:08:23.201387 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:23.201588 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:23.201940 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 02:08:23.202106 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:08:23.206889 29409 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:08:23.213979 29409 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:08:23.251060 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:23.227672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:39879 (local address 127.0.0.1:42058)
0926 02:08:23.238596 (+ 10924us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:23.238679 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:23.239658 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:23.240866 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:23.240872 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:23.240914 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:08:23.241401 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:23.241412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.243301 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.243309 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:23.245825 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:23.245838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.248908 (+ 3070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.248915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:23.250069 (+ 1154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:23.250153 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 02:08:23.250323 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 02:08:23.250518 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10543,"mutex_wait_us":46,"thread_start_us":12134,"threads_started":1}
I20260926 02:08:23.251397 29882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:23.228735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53772 (local address 127.18.159.131:33337)
0926 02:08:23.240659 (+ 11924us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:23.240668 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:23.242266 (+ 1598us) server_negotiation.cc:407] Connection header received
0926 02:08:23.242376 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:23.242383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:23.242631 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:23.242834 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:08:23.243447 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.245114 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.247676 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.249342 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.249554 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:23.249758 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:23.250810 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0926 02:08:23.251034 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11738,"thread_start_us":12381,"threads_started":1}
I20260926 02:08:23.251556 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:23.228359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:40563 (local address 127.0.0.1:52776)
0926 02:08:23.240196 (+ 11837us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:23.240231 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:23.240484 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:23.241541 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:23.241549 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:23.241584 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:08:23.242133 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:23.242144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.244085 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.244095 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:23.246411 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:23.246428 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.250037 (+ 3609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.250135 (+ 98us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:23.250415 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:23.250558 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 02:08:23.250713 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 02:08:23.250904 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11639,"mutex_wait_us":178,"thread_start_us":13391,"threads_started":1}
I20260926 02:08:23.252113 29883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:23.229002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.18.159.130:40563)
0926 02:08:23.239589 (+ 10587us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:23.239603 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:23.240353 (+ 750us) server_negotiation.cc:407] Connection header received
0926 02:08:23.240680 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:23.240691 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:23.240967 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:23.241182 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:08:23.242374 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.243902 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.246649 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.249011 (+ 2362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.250022 (+ 1011us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:23.250270 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:23.251058 (+ 788us) server_negotiation.cc:299] Negotiation successful
0926 02:08:23.251355 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10357,"thread_start_us":10218,"threads_started":1}
I20260926 02:08:23.251907 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:23.228511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:33337 (local address 127.0.0.1:53772)
0926 02:08:23.242100 (+ 13589us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:23.242129 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:23.242256 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:23.242852 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:23.242857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:23.242880 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:08:23.243242 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:23.243257 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.245300 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.245306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:23.247483 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:23.247496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.250258 (+ 2762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.250269 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:23.250532 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:23.250561 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 02:08:23.250690 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 02:08:23.250904 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13449,"mutex_wait_us":66,"thread_start_us":9530,"threads_started":1}
I20260926 02:08:23.252327 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:23.228027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42058 (local address 127.18.159.129:39879)
0926 02:08:23.228752 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:23.228761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:23.238858 (+ 10097us) server_negotiation.cc:407] Connection header received
0926 02:08:23.240002 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:23.240013 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:23.240322 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:23.240525 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:08:23.241599 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.243113 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.246027 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.247897 (+ 1870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.248231 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:23.248749 (+ 518us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:23.251703 (+ 2954us) server_negotiation.cc:299] Negotiation successful
0926 02:08:23.251940 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":621,"threads_started":1}
I20260926 02:08:23.258605 29533 tablet_service.cc:1467] Processing CreateTablet for tablet aacabe13144742158b43fc322bf82a98 (DEFAULT_TABLE table=test-table [id=c82706b47b7a4e8f8022640650143250]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:08:23.258920 29667 tablet_service.cc:1467] Processing CreateTablet for tablet aacabe13144742158b43fc322bf82a98 (DEFAULT_TABLE table=test-table [id=c82706b47b7a4e8f8022640650143250]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:08:23.260366 29533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aacabe13144742158b43fc322bf82a98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:08:23.260530 29667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aacabe13144742158b43fc322bf82a98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:08:23.259902 29801 tablet_service.cc:1467] Processing CreateTablet for tablet aacabe13144742158b43fc322bf82a98 (DEFAULT_TABLE table=test-table [id=c82706b47b7a4e8f8022640650143250]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:08:23.261488 29801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aacabe13144742158b43fc322bf82a98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:08:23.274164 29884 tablet_bootstrap.cc:492] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5: Bootstrap starting.
I20260926 02:08:23.276957 29885 tablet_bootstrap.cc:492] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450: Bootstrap starting.
I20260926 02:08:23.278331 29884 tablet_bootstrap.cc:654] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5: Neither blocks nor log segments found. Creating new log.
I20260926 02:08:23.279336 29886 tablet_bootstrap.cc:492] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b: Bootstrap starting.
I20260926 02:08:23.280287 29884 log.cc:824] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5: Log is configured to *not* fsync() on all Append() calls
I20260926 02:08:23.281608 29885 tablet_bootstrap.cc:654] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450: Neither blocks nor log segments found. Creating new log.
I20260926 02:08:23.283068 29885 log.cc:824] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450: Log is configured to *not* fsync() on all Append() calls
I20260926 02:08:23.284337 29886 tablet_bootstrap.cc:654] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b: Neither blocks nor log segments found. Creating new log.
I20260926 02:08:23.284789 29884 tablet_bootstrap.cc:492] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5: No bootstrap required, opened a new log
I20260926 02:08:23.285112 29884 ts_tablet_manager.cc:1397] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20260926 02:08:23.286031 29886 log.cc:824] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b: Log is configured to *not* fsync() on all Append() calls
I20260926 02:08:23.287221 29885 tablet_bootstrap.cc:492] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450: No bootstrap required, opened a new log
I20260926 02:08:23.289251 29885 ts_tablet_manager.cc:1397] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260926 02:08:23.297921 29884 raft_consensus.cc:348] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e694f452f67c4491b47761eb7c1d2ee5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39879 } } peers { permanent_uuid: "040e77d1f4e84951b3cbd58c68662450" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 40563 } } peers { permanent_uuid: "de4b59f2b8ed4485a3e9b6ba6dfc520b" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 33337 } }
I20260926 02:08:23.298761 29884 raft_consensus.cc:374] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:08:23.298949 29884 raft_consensus.cc:729] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e694f452f67c4491b47761eb7c1d2ee5, State: Initialized, Role: FOLLOWER
I20260926 02:08:23.299777 29884 consensus_queue.cc:260] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e694f452f67c4491b47761eb7c1d2ee5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39879 } } peers { permanent_uuid: "040e77d1f4e84951b3cbd58c68662450" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 40563 } } peers { permanent_uuid: "de4b59f2b8ed4485a3e9b6ba6dfc520b" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 33337 } }
I20260926 02:08:23.301887 29885 raft_consensus.cc:348] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e694f452f67c4491b47761eb7c1d2ee5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39879 } } peers { permanent_uuid: "040e77d1f4e84951b3cbd58c68662450" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 40563 } } peers { permanent_uuid: "de4b59f2b8ed4485a3e9b6ba6dfc520b" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 33337 } }
I20260926 02:08:23.302425 29885 raft_consensus.cc:374] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:08:23.302544 29885 raft_consensus.cc:729] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040e77d1f4e84951b3cbd58c68662450, State: Initialized, Role: FOLLOWER
I20260926 02:08:23.303318 29886 tablet_bootstrap.cc:492] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b: No bootstrap required, opened a new log
I20260926 02:08:23.303480 29885 consensus_queue.cc:260] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e694f452f67c4491b47761eb7c1d2ee5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39879 } } peers { permanent_uuid: "040e77d1f4e84951b3cbd58c68662450" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 40563 } } peers { permanent_uuid: "de4b59f2b8ed4485a3e9b6ba6dfc520b" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 33337 } }
I20260926 02:08:23.303869 29884 ts_tablet_manager.cc:1428] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5: Time spent starting tablet: real 0.019s user 0.015s sys 0.002s
I20260926 02:08:23.303854 29886 ts_tablet_manager.cc:1397] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.008s
I20260926 02:08:23.305351 29732 heartbeater.cc:503] Master 127.18.159.190:38437 was elected leader, sending a full tablet report...
I20260926 02:08:23.306241 29885 ts_tablet_manager.cc:1428] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450: Time spent starting tablet: real 0.017s user 0.008s sys 0.007s
I20260926 02:08:23.316660 29886 raft_consensus.cc:348] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e694f452f67c4491b47761eb7c1d2ee5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39879 } } peers { permanent_uuid: "040e77d1f4e84951b3cbd58c68662450" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 40563 } } peers { permanent_uuid: "de4b59f2b8ed4485a3e9b6ba6dfc520b" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 33337 } }
I20260926 02:08:23.317566 29886 raft_consensus.cc:374] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:08:23.317725 29886 raft_consensus.cc:729] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de4b59f2b8ed4485a3e9b6ba6dfc520b, State: Initialized, Role: FOLLOWER
I20260926 02:08:23.318601 29886 consensus_queue.cc:260] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e694f452f67c4491b47761eb7c1d2ee5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39879 } } peers { permanent_uuid: "040e77d1f4e84951b3cbd58c68662450" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 40563 } } peers { permanent_uuid: "de4b59f2b8ed4485a3e9b6ba6dfc520b" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 33337 } }
I20260926 02:08:23.319821 29866 heartbeater.cc:503] Master 127.18.159.190:38437 was elected leader, sending a full tablet report...
I20260926 02:08:23.320569 29886 ts_tablet_manager.cc:1428] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b: Time spent starting tablet: real 0.017s user 0.003s sys 0.013s
W20260926 02:08:23.332580 29599 tablet.cc:2369] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:08:23.371215 29733 tablet.cc:2369] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:08:23.387137 29891 raft_consensus.cc:482] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:08:23.387509 29891 raft_consensus.cc:504] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e694f452f67c4491b47761eb7c1d2ee5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39879 } } peers { permanent_uuid: "040e77d1f4e84951b3cbd58c68662450" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 40563 } } peers { permanent_uuid: "de4b59f2b8ed4485a3e9b6ba6dfc520b" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 33337 } }
I20260926 02:08:23.389415 29891 leader_election.cc:288] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e694f452f67c4491b47761eb7c1d2ee5 (127.18.159.129:39879), de4b59f2b8ed4485a3e9b6ba6dfc520b (127.18.159.131:33337)
I20260926 02:08:23.398068 29882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:23.389975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:51575 (local address 127.18.159.131:33337)
0926 02:08:23.390235 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:23.390242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:23.391154 (+ 912us) server_negotiation.cc:407] Connection header received
0926 02:08:23.391404 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:23.391408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:23.391469 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:23.391572 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:08:23.392302 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.393243 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.395965 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.396954 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.397185 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:23.397272 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:23.397644 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 02:08:23.397778 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 02:08:23.398234 29894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:23.389689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:33337 (local address 127.18.159.130:51575)
0926 02:08:23.391003 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:23.391021 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:23.391231 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:23.391606 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:23.391611 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:23.391636 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:08:23.392083 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:23.392094 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.393401 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.393406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:23.395723 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:23.395741 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.397249 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.397256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:23.397430 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:23.397456 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 02:08:23.397609 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 02:08:23.397778 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1174,"mutex_wait_us":32,"thread_start_us":757,"threads_started":1}
I20260926 02:08:23.398425 29893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:23.389842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:39879 (local address 127.18.159.130:39183)
0926 02:08:23.391049 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0926 02:08:23.391065 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:08:23.391306 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:08:23.391678 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:08:23.391682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:08:23.391706 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:08:23.392238 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:23.392256 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.393479 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.393488 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:23.395778 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:08:23.395793 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.397495 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.397502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:08:23.397680 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:23.397722 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 02:08:23.397850 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 02:08:23.397999 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"mutex_wait_us":129,"thread_start_us":459,"threads_started":1}
I20260926 02:08:23.398739 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:23.389792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:39183 (local address 127.18.159.129:39879)
0926 02:08:23.390006 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:23.390014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:23.391208 (+ 1194us) server_negotiation.cc:407] Connection header received
0926 02:08:23.391460 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:23.391464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:23.391523 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:23.391654 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:08:23.392467 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.393307 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.396140 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.397229 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.397472 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:23.397545 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:23.397875 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 02:08:23.398001 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:08:23.399911 29553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacabe13144742158b43fc322bf82a98" candidate_uuid: "040e77d1f4e84951b3cbd58c68662450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e694f452f67c4491b47761eb7c1d2ee5" is_pre_election: true
I20260926 02:08:23.399904 29821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacabe13144742158b43fc322bf82a98" candidate_uuid: "040e77d1f4e84951b3cbd58c68662450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de4b59f2b8ed4485a3e9b6ba6dfc520b" is_pre_election: true
W20260926 02:08:23.400583 29867 tablet.cc:2369] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:08:23.400539 29553 raft_consensus.cc:2454] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 040e77d1f4e84951b3cbd58c68662450 in term 0.
I20260926 02:08:23.400556 29821 raft_consensus.cc:2454] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 040e77d1f4e84951b3cbd58c68662450 in term 0.
I20260926 02:08:23.401376 29620 leader_election.cc:302] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 040e77d1f4e84951b3cbd58c68662450, e694f452f67c4491b47761eb7c1d2ee5; no voters:
I20260926 02:08:23.401845 29891 raft_consensus.cc:2781] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:08:23.402030 29891 raft_consensus.cc:482] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:08:23.402148 29891 raft_consensus.cc:3037] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:08:23.404462 29891 raft_consensus.cc:504] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e694f452f67c4491b47761eb7c1d2ee5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39879 } } peers { permanent_uuid: "040e77d1f4e84951b3cbd58c68662450" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 40563 } } peers { permanent_uuid: "de4b59f2b8ed4485a3e9b6ba6dfc520b" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 33337 } }
I20260926 02:08:23.405486 29891 leader_election.cc:288] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 [CANDIDATE]: Term 1 election: Requested vote from peers e694f452f67c4491b47761eb7c1d2ee5 (127.18.159.129:39879), de4b59f2b8ed4485a3e9b6ba6dfc520b (127.18.159.131:33337)
I20260926 02:08:23.406172 29553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacabe13144742158b43fc322bf82a98" candidate_uuid: "040e77d1f4e84951b3cbd58c68662450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e694f452f67c4491b47761eb7c1d2ee5"
I20260926 02:08:23.406212 29821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacabe13144742158b43fc322bf82a98" candidate_uuid: "040e77d1f4e84951b3cbd58c68662450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de4b59f2b8ed4485a3e9b6ba6dfc520b"
I20260926 02:08:23.406419 29821 raft_consensus.cc:3037] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:08:23.406456 29553 raft_consensus.cc:3037] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:08:23.409974 29821 raft_consensus.cc:2454] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 040e77d1f4e84951b3cbd58c68662450 in term 1.
I20260926 02:08:23.409981 29553 raft_consensus.cc:2454] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 040e77d1f4e84951b3cbd58c68662450 in term 1.
I20260926 02:08:23.410759 29620 leader_election.cc:302] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 040e77d1f4e84951b3cbd58c68662450, e694f452f67c4491b47761eb7c1d2ee5; no voters:
I20260926 02:08:23.411201 29891 raft_consensus.cc:2781] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:08:23.411692 29891 raft_consensus.cc:686] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 [term 1 LEADER]: Becoming Leader. State: Replica: 040e77d1f4e84951b3cbd58c68662450, State: Running, Role: LEADER
I20260926 02:08:23.412237 29891 consensus_queue.cc:237] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e694f452f67c4491b47761eb7c1d2ee5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39879 } } peers { permanent_uuid: "040e77d1f4e84951b3cbd58c68662450" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 40563 } } peers { permanent_uuid: "de4b59f2b8ed4485a3e9b6ba6dfc520b" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 33337 } }
I20260926 02:08:23.417548 29408 catalog_manager.cc:5495] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 reported cstate change: term changed from 0 to 1, leader changed from <none> to 040e77d1f4e84951b3cbd58c68662450 (127.18.159.130). New cstate: current_term: 1 leader_uuid: "040e77d1f4e84951b3cbd58c68662450" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e694f452f67c4491b47761eb7c1d2ee5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "040e77d1f4e84951b3cbd58c68662450" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 40563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de4b59f2b8ed4485a3e9b6ba6dfc520b" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 33337 } health_report { overall_health: UNKNOWN } } }
I20260926 02:08:23.451222 29883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:23.445431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.18.159.130:40563)
0926 02:08:23.445701 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:23.445708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:23.445821 (+ 113us) server_negotiation.cc:407] Connection header received
0926 02:08:23.446104 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:23.446109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:23.446187 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:23.446365 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:08:23.446759 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.447551 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.449028 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.449427 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.449572 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:23.450098 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:08:23.450607 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:08:23.450792 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:23.450851 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 02:08:23.450944 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 02:08:23.460763 29553 raft_consensus.cc:1261] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5 [term 1 FOLLOWER]: Refusing update from remote peer 040e77d1f4e84951b3cbd58c68662450: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:08:23.460753 29821 raft_consensus.cc:1261] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b [term 1 FOLLOWER]: Refusing update from remote peer 040e77d1f4e84951b3cbd58c68662450: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:08:23.461990 29891 consensus_queue.cc:1035] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e694f452f67c4491b47761eb7c1d2ee5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:08:23.462419 29895 consensus_queue.cc:1035] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de4b59f2b8ed4485a3e9b6ba6dfc520b" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 33337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:08:23.475185 29898 mvcc.cc:205] Tried to move back new op lower bound from 7333431310163283968 to 7333431309980389376. Current Snapshot: MvccSnapshot[applied={T|T < 7333431310163283968}]
I20260926 02:08:23.491261 29408 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:08:23.491607 29408 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:08:23.495626 29408 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 513ecca5256248f9ba4336608e4aaef5: Sending DeleteTablet for 3 replicas of tablet aacabe13144742158b43fc322bf82a98
I20260926 02:08:23.496905 29533 tablet_service.cc:1514] Processing DeleteTablet for tablet aacabe13144742158b43fc322bf82a98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:08:23 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:42058
I20260926 02:08:23.496937 29801 tablet_service.cc:1514] Processing DeleteTablet for tablet aacabe13144742158b43fc322bf82a98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:08:23 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:53772
I20260926 02:08:23.497772 29667 tablet_service.cc:1514] Processing DeleteTablet for tablet aacabe13144742158b43fc322bf82a98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:08:23 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:52776
I20260926 02:08:23.498261 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 29470
I20260926 02:08:23.504860 29910 tablet_replica.cc:330] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450: stopping tablet replica
I20260926 02:08:23.505412 29910 raft_consensus.cc:2229] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:08:23.506156 29910 raft_consensus.cc:2258] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:08:23.509406 29910 ts_tablet_manager.cc:1905] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:08:23.506266 29909 tablet_replica.cc:330] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b: stopping tablet replica
I20260926 02:08:23.510269 29908 tablet_replica.cc:330] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5: stopping tablet replica
I20260926 02:08:23.510562 29909 raft_consensus.cc:2229] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:08:23.510712 29908 raft_consensus.cc:2229] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:08:23.511224 29908 raft_consensus.cc:2258] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:08:23.511224 29909 raft_consensus.cc:2258] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:08:23.513646 29909 ts_tablet_manager.cc:1905] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:08:23.517413 29908 ts_tablet_manager.cc:1905] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:08:23.517843 29910 ts_tablet_manager.cc:1918] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:08:23.518031 29910 log.cc:1196] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/aacabe13144742158b43fc322bf82a98
I20260926 02:08:23.519022 29910 ts_tablet_manager.cc:1939] T aacabe13144742158b43fc322bf82a98 P 040e77d1f4e84951b3cbd58c68662450: Deleting consensus metadata
Sep 26 02:08:23 dist-test-slave-03rn krb5kdc[29364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:08:23.521771 29909 ts_tablet_manager.cc:1918] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:08:23.522055 29909 log.cc:1196] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/aacabe13144742158b43fc322bf82a98
I20260926 02:08:23.522647 29909 ts_tablet_manager.cc:1939] T aacabe13144742158b43fc322bf82a98 P de4b59f2b8ed4485a3e9b6ba6dfc520b: Deleting consensus metadata
I20260926 02:08:23.524915 29394 catalog_manager.cc:4841] TS 040e77d1f4e84951b3cbd58c68662450 (127.18.159.130:40563): tablet aacabe13144742158b43fc322bf82a98 (table test-table [id=c82706b47b7a4e8f8022640650143250]) successfully deleted
I20260926 02:08:23.524938 29395 catalog_manager.cc:4841] TS de4b59f2b8ed4485a3e9b6ba6dfc520b (127.18.159.131:33337): tablet aacabe13144742158b43fc322bf82a98 (table test-table [id=c82706b47b7a4e8f8022640650143250]) successfully deleted
I20260926 02:08:23.526268 29908 ts_tablet_manager.cc:1918] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:08:23.526476 29908 log.cc:1196] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/aacabe13144742158b43fc322bf82a98
I20260926 02:08:23.527017 29908 ts_tablet_manager.cc:1939] T aacabe13144742158b43fc322bf82a98 P e694f452f67c4491b47761eb7c1d2ee5: Deleting consensus metadata
I20260926 02:08:23.529070 29393 catalog_manager.cc:4841] TS e694f452f67c4491b47761eb7c1d2ee5 (127.18.159.129:39879): tablet aacabe13144742158b43fc322bf82a98 (table test-table [id=c82706b47b7a4e8f8022640650143250]) successfully deleted
I20260926 02:08:23.530900 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:23.499766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.18.159.129:39879)
0926 02:08:23.500042 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:23.500051 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:23.507414 (+ 7363us) server_negotiation.cc:407] Connection header received
0926 02:08:23.507754 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:23.507760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:23.507846 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:23.507926 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:23.511655 (+ 3729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.512840 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.515156 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.515780 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.515914 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:23.522644 (+ 6730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:23.522661 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:23.522680 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:08:23.522735 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:23.527444 (+ 4709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:08:23.528543 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:08:23.528554 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:08:23.528557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:08:23.528732 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:08:23.529205 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:08:23.529210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:08:23.529213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:08:23.529655 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:23.529983 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:23.530310 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 02:08:23.530525 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 02:08:23.531564 29593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:08:23.626341 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 29470
I20260926 02:08:23.652439 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 29604
Sep 26 02:08:23 dist-test-slave-03rn krb5kdc[29364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.130@KRBTEST.COM
I20260926 02:08:23.672621 29883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:23.653439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52788 (local address 127.18.159.130:40563)
0926 02:08:23.653664 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:23.653671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:23.653695 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:08:23.653806 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:23.653812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:23.653905 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:23.654034 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:23.655372 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.656900 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.658927 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.659738 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.659904 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:23.665305 (+ 5401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:23.665323 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:23.665340 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:08:23.665386 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:23.668984 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:08:23.670006 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:08:23.670015 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:08:23.670018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:08:23.670187 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:08:23.670696 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:08:23.670701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:08:23.670703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:08:23.671135 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:23.671521 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:23.671868 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 02:08:23.672135 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:08:23.673493 29727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:08:23.789639 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 29604
I20260926 02:08:23.808454 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 29738
Sep 26 02:08:23 dist-test-slave-03rn krb5kdc[29364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.131@KRBTEST.COM
I20260926 02:08:23.824867 29882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:08:23.809233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53784 (local address 127.18.159.131:33337)
0926 02:08:23.809442 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 02:08:23.809448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:08:23.809469 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:08:23.809673 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:08:23.809679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:08:23.809763 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:08:23.809883 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:08:23.811162 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.811890 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.813225 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:08:23.813648 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:08:23.813797 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:08:23.818515 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:08:23.818533 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:08:23.818549 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:08:23.818592 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:08:23.821715 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:08:23.822558 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:08:23.822572 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:08:23.822575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:08:23.822716 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:08:23.823129 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:08:23.823141 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:08:23.823144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:08:23.823527 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:08:23.823851 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0926 02:08:23.824207 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 02:08:23.824447 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:08:23.825599 29861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:08:23.915472 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 29738
I20260926 02:08:23.932324 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 29377
I20260926 02:08:23.933513 29439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:08:24.042044 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 29377
2026-09-26T02:08:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4526 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:08:24 dist-test-slave-03rn krb5kdc[29918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:08:24 dist-test-slave-03rn krb5kdc[29918](info): set up 2 sockets
Sep 26 02:08:24 dist-test-slave-03rn krb5kdc[29918](info): commencing operation
krb5kdc: starting...
W20260926 02:08:25.181617 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:08:25 dist-test-slave-03rn krb5kdc[29918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:08:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:08:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.159.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.159.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.159.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/rangeradmin_127.18.159.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.159.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/rangeradmin_127.18.159.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.159.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.159.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.159.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/rangerlookup_127.18.159.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.159.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/rangerlookup_127.18.159.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.159.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.159.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.159.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/HTTP_127.18.159.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.159.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/HTTP_127.18.159.148@KRBTEST.COM.keytab.
I20260926 02:08:25.535573 19070 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8FmYUC/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/postgres -l logfile start
2026-09-26 02:08:28.365 UTC [29948] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 02:08:28.366 UTC [29948] LOG: listening on IPv4 address "127.18.159.148", port 55129
2026-09-26 02:08:28.370 UTC [29948] LOG: listening on Unix socket "/tmp/.s.PGSQL.55129"
2026-09-26 02:08:28.394 UTC [29951] LOG: database system was shut down at 2026-09-26 02:08:26 UTC
2026-09-26 02:08:28.398 UTC [29948] LOG: database system is ready to accept connections
I20260926 02:08:29.385265 19070 mini_postgres.cc:96] Postgres bound to 55129
2026-09-26 02:08:29.404 UTC [29959] FATAL: database "slave" does not exist
127.18.159.148:55129 - accepting connections
I20260926 02:08:29.405218 19070 mini_ranger.cc:161] Starting Ranger...
I20260926 02:08:29.434312 19070 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 02:08:30.011955 19070 mini_ranger.cc:87] Created ranger Postgres database
I20260926 02:08:30.012108 19070 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ranger-admin
2026-09-26 02:08:30,332 [I] DB FLAVOR :POSTGRES
2026-09-26 02:08:30,332 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 02:08:30,333 [I] Checking connection..
2026-09-26 02:08:30,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 02:08:30,612 [I] Checking connection passed.
2026-09-26 02:08:30,613 [I] --------- Verifying version history table ---------
2026-09-26 02:08:30,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:08:30,861 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 02:08:30,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:08:31,123 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 02:08:31,124 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 02:08:31,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 02:08:31.387 UTC [30023] WARNING: there is no transaction in progress
2026-09-26 02:08:31,394 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 02:08:31,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:08:31,663 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 02:08:31,664 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 02:08:31,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 02:08:31,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 02:08:32,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-03rn.c.gcp-upstream.internal','N') ;"
2026-09-26 02:08:32,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 02:08:32,706 [I] Table x_portal_user does not exist in database ranger
2026-09-26 02:08:32,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 02:08:32,950 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 02:08:32,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 02:08:33,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 02:08:33,442 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 02:08:33,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 02:08:34.486 UTC [30131] WARNING: there is no transaction in progress
2026-09-26 02:08:34.504 UTC [30131] WARNING: there is no transaction in progress
2026-09-26 02:08:34.524 UTC [30131] WARNING: there is no transaction in progress
2026-09-26 02:08:34.546 UTC [30131] WARNING: there is no transaction in progress
2026-09-26 02:08:34.567 UTC [30131] WARNING: there is no transaction in progress
2026-09-26 02:08:34.588 UTC [30131] WARNING: there is no transaction in progress
2026-09-26 02:08:34.661 UTC [30131] WARNING: there is no transaction in progress
2026-09-26 02:08:34.671 UTC [30131] WARNING: there is no transaction in progress
2026-09-26 02:08:34.692 UTC [30131] WARNING: there is no transaction in progress
2026-09-26 02:08:34.707 UTC [30131] WARNING: there is no transaction in progress
2026-09-26 02:08:34.721 UTC [30131] WARNING: there is no transaction in progress
2026-09-26 02:08:34.739 UTC [30131] WARNING: there is no transaction in progress
2026-09-26 02:08:34.751 UTC [30131] WARNING: there is no transaction in progress
2026-09-26 02:08:34.760 UTC [30131] WARNING: there is no transaction in progress
2026-09-26 02:08:34.770 UTC [30131] WARNING: there is no transaction in progress
2026-09-26 02:08:34.778 UTC [30131] WARNING: there is no transaction in progress
2026-09-26 02:08:35,549 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 02:08:35,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 02:08:35,798 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 02:08:35,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 02:08:36,042 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 02:08:36,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 02:08:36,290 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 02:08:36,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 02:08:36,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 02:08:36,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-03rn.c.gcp-upstream.internal';"
2026-09-26 02:08:37,042 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 02:08:37,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 02:08:37,283 [I] DB_PATCHES have already been applied
I20260926 02:08:37.289186 19070 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ranger-admin:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8FmYUC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 02:08:37.289320 19070 mini_ranger.cc:193] Using host: 127.18.159.148
I20260926 02:08:37.305163 19070 mini_ranger.cc:227] Ranger admin URL: http://127.18.159.148:56323
Sep 26, 2026 2:08:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:08:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:08:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 2:08:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 2:08:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 2:08:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 2:08:38 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 02:08:38.472882 19070 mini_ranger.cc:160] Time spent starting Ranger: real 9.068s user 0.006s sys 0.039s
Sep 26, 2026 2:08:59 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 2:08:59 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 2:08:59 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 2:08:59 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 2:08:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 2:08:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 2:08:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 2:08:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 2:08:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 2:08:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 2:08:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 2:08:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 2:08:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 2:08:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 2:08:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 2:08:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 2:08:59 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 2:09:00 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 02:09:01.544361 19070 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.159.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.159.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.159.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/rangerkms_127.18.159.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.159.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/rangerkms_127.18.159.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.159.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.159.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.159.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/HTTP_127.18.159.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.159.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/HTTP_127.18.159.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 02:09:01 dist-test-slave-03rn krb5kdc[29918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388541, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 02:09:01.726692 19070 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 02:09:01.749662 19070 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 02:09:02.639496 19070 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 02:09:02.639662 19070 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ranger-kms
I20260926 02:09:02.639810 19070 mini_ranger_kms.cc:227] Using postgres at 127.18.159.148:55129
2026-09-26 02:09:03,080 [I] DB FLAVOR :POSTGRES
2026-09-26 02:09:03,081 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 02:09:03,081 [I] Checking connection
2026-09-26 02:09:03,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 02:09:03,322 [I] connection success
2026-09-26 02:09:03,322 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 02:09:03,322 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 02:09:03,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 02:09:03,592 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 02:09:03,592 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 02:09:03,592 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 02:09:03,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:55129/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 02:09:03,866 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 02:09:04.467047 19070 mini_ranger_kms.cc:310] Created kms service
I20260926 02:09:04.674813 19070 mini_ranger_kms.cc:326] Created kudu user
I20260926 02:09:04.725494 19070 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 02:09:04 dist-test-slave-03rn krb5kdc[29918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388541, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
I20260926 02:09:05.134527 19070 mini_ranger_kms.cc:384] Added ranger policy
I20260926 02:09:05.134843 19070 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ranger-kms:/tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 02:09:05.134927 19070 mini_ranger_kms.cc:240] Using host: 127.18.159.148
I20260926 02:09:05.157150 19070 mini_ranger_kms.cc:276] Ranger KMS PID: 30312
I20260926 02:09:05.157385 19070 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.159.148:54103
Sep 26, 2026 2:09:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:09:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:09:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 2:09:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 2:09:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 2:09:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 02:09:07.571727 19070 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.845s user 0.125s sys 0.339s
I20260926 02:09:07.571974 19070 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 02:09:07.572203 19070 mini_ranger_kms.cc:401] 127.18.159.148:54103/kms/v1/keys
Sep 26 02:09:07 dist-test-slave-03rn krb5kdc[29918](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790388547, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.159.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 02:09:08 dist-test-slave-03rn krb5kdc[29918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388547, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.159.148@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
Sep 26 02:09:08 dist-test-slave-03rn krb5kdc[29918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388547, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.159.148@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
Sep 26 02:09:10 dist-test-slave-03rn krb5kdc[29918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:09:10.917548 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:33123
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:43009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:33123
--ranger_config_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.159.148:54103/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:09:11.296440 30362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:09:11.296753 30362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:09:11.296905 30362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:09:11.305013 30362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:09:11.305151 30362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:09:11.305225 30362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:09:11.305311 30362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:09:11.305356 30362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:09:11.316013 30362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:43009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.159.148:54103/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:33123
--ranger_config_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:33123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:09:11.317517 30362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:09:11.318929 30362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:09:11.332314 30368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:09:11.332420 30367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:09:11.332396 30370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:09:11.333206 30362 server_base.cc:1034] running on GCE node
I20260926 02:09:11.334075 30362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:09:11.335623 30362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:09:11.336856 30362 hybrid_clock.cc:648] HybridClock initialized: now 1790388551336814 us; error 43 us; skew 500 ppm
I20260926 02:09:11.337306 30362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:09:11 dist-test-slave-03rn krb5kdc[29918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388551, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:09:11.344630 30362 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:09:11.346319 30362 webserver.cc:460] Webserver started at http://127.18.159.190:44723/ using document root <none> and password file <none>
I20260926 02:09:11.347141 30362 fs_manager.cc:359] Metadata directory not provided
I20260926 02:09:11.347296 30362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:09:11.347702 30362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:09:11 dist-test-slave-03rn krb5kdc[29918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388551, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
I20260926 02:09:11.400982 30362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "28d81752911f44d0be55457aecca5ac2"
format_stamp: "Formatted at 2026-09-26 02:09:11 on dist-test-slave-03rn"
server_key: "a4068cd7c39e7b86800ad4fb760ced9c"
server_key_iv: "2ebe09c1a7ebb60065ed6f9a78f4b852"
server_key_version: "kuduclusterkey@0"
I20260926 02:09:11.401919 30362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "28d81752911f44d0be55457aecca5ac2"
format_stamp: "Formatted at 2026-09-26 02:09:11 on dist-test-slave-03rn"
server_key: "a4068cd7c39e7b86800ad4fb760ced9c"
server_key_iv: "2ebe09c1a7ebb60065ed6f9a78f4b852"
server_key_version: "kuduclusterkey@0"
I20260926 02:09:11.408047 30362 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260926 02:09:11.425225 30377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:09:11.426944 30362 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:09:11.427172 30362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "28d81752911f44d0be55457aecca5ac2"
format_stamp: "Formatted at 2026-09-26 02:09:11 on dist-test-slave-03rn"
server_key: "a4068cd7c39e7b86800ad4fb760ced9c"
server_key_iv: "2ebe09c1a7ebb60065ed6f9a78f4b852"
server_key_version: "kuduclusterkey@0"
I20260926 02:09:11.427412 30362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:09:11.448370 30362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:09:11.452800 30362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:09:11.453150 30362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:09:11.475570 30362 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:33123
I20260926 02:09:11.475589 30428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:33123 every 8 connection(s)
I20260926 02:09:11.477129 30362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:09:11.480507 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 30362
I20260926 02:09:11.480710 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:09:11.481449 19070 external_mini_cluster.cc:1428] Setting key 8e2ca6fde9b451acaa20fed15c26c7b6
I20260926 02:09:11.482049 30429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:09:11.501621 30429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2: Bootstrap starting.
I20260926 02:09:11.506057 30429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2: Neither blocks nor log segments found. Creating new log.
I20260926 02:09:11.507500 30429 log.cc:824] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2: Log is configured to *not* fsync() on all Append() calls
I20260926 02:09:11.511381 30429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2: No bootstrap required, opened a new log
Sep 26 02:09:11 dist-test-slave-03rn krb5kdc[29918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:09:11.522720 30429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d81752911f44d0be55457aecca5ac2" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 33123 } }
I20260926 02:09:11.523321 30429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:09:11.523437 30429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28d81752911f44d0be55457aecca5ac2, State: Initialized, Role: FOLLOWER
I20260926 02:09:11.524257 30429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d81752911f44d0be55457aecca5ac2" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 33123 } }
I20260926 02:09:11.524542 30429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:09:11.524696 30429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:09:11.524887 30429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:09:11.524935 30432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:11.483624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40754 (local address 127.18.159.190:33123)
0926 02:09:11.496806 (+ 13182us) server_negotiation.cc:206] Beginning negotiation
0926 02:09:11.496824 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 02:09:11.497862 (+ 1038us) server_negotiation.cc:407] Connection header received
0926 02:09:11.500465 (+ 2603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:09:11.500501 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:09:11.501114 (+ 613us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:09:11.501618 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:09:11.503375 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:11.505949 (+ 2574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:11.507896 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:11.509022 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:11.509223 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:11.514341 (+ 5118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:09:11.514354 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:09:11.514388 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:09:11.514458 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:09:11.519296 (+ 4838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:09:11.520408 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:09:11.520420 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:09:11.520443 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:09:11.520675 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:09:11.521138 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:09:11.521144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:09:11.521148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:09:11.522013 (+ 865us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:09:11.522315 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 02:09:11.522861 (+ 546us) server_negotiation.cc:299] Negotiation successful
0926 02:09:11.523411 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12915,"thread_start_us":12663,"threads_started":1}
I20260926 02:09:11.528069 30429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d81752911f44d0be55457aecca5ac2" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 33123 } }
I20260926 02:09:11.528784 30429 leader_election.cc:302] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28d81752911f44d0be55457aecca5ac2; no voters:
I20260926 02:09:11.529577 30429 leader_election.cc:288] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:09:11.529686 30434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:09:11.532603 30429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:09:11.537283 30434 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2 [term 1 LEADER]: Becoming Leader. State: Replica: 28d81752911f44d0be55457aecca5ac2, State: Running, Role: LEADER
I20260926 02:09:11.538172 30434 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d81752911f44d0be55457aecca5ac2" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 33123 } }
I20260926 02:09:11.548043 30436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28d81752911f44d0be55457aecca5ac2. Latest consensus state: current_term: 1 leader_uuid: "28d81752911f44d0be55457aecca5ac2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d81752911f44d0be55457aecca5ac2" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 33123 } } }
I20260926 02:09:11.548532 30436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2 [sys.catalog]: This master's current role is: LEADER
I20260926 02:09:11.549067 30435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28d81752911f44d0be55457aecca5ac2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d81752911f44d0be55457aecca5ac2" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 33123 } } }
I20260926 02:09:11.549333 30435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2 [sys.catalog]: This master's current role is: LEADER
I20260926 02:09:11.552598 30445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:09:11.560660 30445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:09:11.563231 30429 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 02:09:11.584522 30445 catalog_manager.cc:1347] Generated new cluster ID: abbfef71c4a74e589c6c6fa701f0b5aa
I20260926 02:09:11.584760 30445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:09:11.604756 30445 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:09:11.606998 30445 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:09:11.627701 30445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2: Generated new TSK 0
I20260926 02:09:11.629136 30445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.159.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
I20260926 02:09:12.433169 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.159.148:54103/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:33123
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:43009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:09:12 dist-test-slave-03rn krb5kdc[29918](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790388552, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 02:09:12.865594 30475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:09:12.865944 30475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:09:12.866118 30475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:09:12.874229 30475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:09:12.874398 30475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:09:12.874593 30475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:09:12.887548 30475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:43009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.159.148:54103/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:33123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:09:12.889073 30475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:09:12.890514 30475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:09:12.910147 30485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:09:12.910147 30486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:09:12.913518 30475 server_base.cc:1034] running on GCE node
W20260926 02:09:12.914167 30489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:09:12.915079 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:09:12.916716 30475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:09:12.917989 30475 hybrid_clock.cc:648] HybridClock initialized: now 1790388552917925 us; error 67 us; skew 500 ppm
I20260926 02:09:12.918434 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:09:12 dist-test-slave-03rn krb5kdc[29918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388552, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:09:12.922801 30475 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:09:12.925824 30475 webserver.cc:460] Webserver started at http://127.18.159.129:45183/ using document root <none> and password file <none>
I20260926 02:09:12.926884 30475 fs_manager.cc:359] Metadata directory not provided
I20260926 02:09:12.927054 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:09:12.927474 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:09:12 dist-test-slave-03rn krb5kdc[29918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388552, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
I20260926 02:09:12.955829 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "c6ed22dcbf9f41ab90fbc26e6cec480b"
format_stamp: "Formatted at 2026-09-26 02:09:12 on dist-test-slave-03rn"
server_key: "066e631c40a8835fffbe27a21ce6ddde"
server_key_iv: "c8eb2a25fc85915edd8e7ca17c8bf15b"
server_key_version: "kuduclusterkey@0"
I20260926 02:09:12.958626 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "c6ed22dcbf9f41ab90fbc26e6cec480b"
format_stamp: "Formatted at 2026-09-26 02:09:12 on dist-test-slave-03rn"
server_key: "066e631c40a8835fffbe27a21ce6ddde"
server_key_iv: "c8eb2a25fc85915edd8e7ca17c8bf15b"
server_key_version: "kuduclusterkey@0"
I20260926 02:09:12.967190 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.000s
I20260926 02:09:13.004956 30495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:09:13.007225 30475 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.008s sys 0.012s
I20260926 02:09:13.007666 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "c6ed22dcbf9f41ab90fbc26e6cec480b"
format_stamp: "Formatted at 2026-09-26 02:09:12 on dist-test-slave-03rn"
server_key: "066e631c40a8835fffbe27a21ce6ddde"
server_key_iv: "c8eb2a25fc85915edd8e7ca17c8bf15b"
server_key_version: "kuduclusterkey@0"
I20260926 02:09:13.008047 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:09:13.034025 30475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:09:13.038700 30475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:09:13.039326 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:09:13.040520 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:09:13.042045 30475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:09:13.042158 30475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:09:13.042320 30475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:09:13.042404 30475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:09:13.148564 30475 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:37469
I20260926 02:09:13.148604 30607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:37469 every 8 connection(s)
I20260926 02:09:13.150498 30475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:09:13.151525 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 30475
I20260926 02:09:13.151770 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:09:13.152590 19070 external_mini_cluster.cc:1428] Setting key 2c4449366a82a975d5940d8836ccf7f4
Sep 26 02:09:13 dist-test-slave-03rn krb5kdc[29918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388552, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.159.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.130@KRBTEST.COM" created.
I20260926 02:09:13.197840 30611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:13.161317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:54015 (local address 127.18.159.190:33123)
0926 02:09:13.164870 (+ 3553us) server_negotiation.cc:206] Beginning negotiation
0926 02:09:13.164877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:09:13.171325 (+ 6448us) server_negotiation.cc:407] Connection header received
0926 02:09:13.171408 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:09:13.171414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:09:13.171512 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:09:13.171640 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:09:13.174068 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:13.176764 (+ 2696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:13.179683 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:13.180389 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:13.180541 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:13.186335 (+ 5794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:09:13.186345 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:09:13.186350 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:09:13.186396 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:09:13.190713 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:09:13.191926 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:09:13.191933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:09:13.191937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:09:13.192091 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:09:13.192722 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:09:13.192727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:09:13.192730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:09:13.193119 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:09:13.193259 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 02:09:13.196665 (+ 3406us) server_negotiation.cc:299] Negotiation successful
0926 02:09:13.196972 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3409,"thread_start_us":3271,"threads_started":1}
I20260926 02:09:13.198408 30610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:13.160617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:33123 (local address 127.18.159.129:54015)
0926 02:09:13.167811 (+ 7194us) negotiation.cc:107] Waiting for socket to connect
0926 02:09:13.167900 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0926 02:09:13.170849 (+ 2949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:09:13.172018 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:09:13.172034 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:09:13.172754 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:09:13.173801 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:09:13.173832 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:09:13.177035 (+ 3203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:09:13.177042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:09:13.179434 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:09:13.179457 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:09:13.180895 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:09:13.180903 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:09:13.181219 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:13.182138 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:09:13.182172 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:09:13.185973 (+ 3801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:09:13.190990 (+ 5017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:09:13.191001 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:09:13.191022 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:09:13.191753 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:09:13.192270 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:09:13.192286 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:09:13.192290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:09:13.192562 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:09:13.193366 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:09:13.193376 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:09:13.193899 (+ 523us) client_negotiation.cc:769] Sending connection context
0926 02:09:13.194250 (+ 351us) client_negotiation.cc:240] Negotiation successful
0926 02:09:13.194854 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6844,"thread_start_us":6589,"threads_started":1}
I20260926 02:09:13.200819 30608 heartbeater.cc:347] Connected to a master server at 127.18.159.190:33123
I20260926 02:09:13.201184 30608 heartbeater.cc:464] Registering TS with master...
I20260926 02:09:13.202102 30608 heartbeater.cc:511] Master 127.18.159.190:33123 requested a full tablet report, sending...
I20260926 02:09:13.204921 30394 ts_manager.cc:194] Registered new tserver with Master: c6ed22dcbf9f41ab90fbc26e6cec480b (127.18.159.129:37469)
I20260926 02:09:13.207123 30394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:54015
Sep 26 02:09:13 dist-test-slave-03rn krb5kdc[29918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388552, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
I20260926 02:09:13.243360 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.159.148:54103/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:33123
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:43009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:09:13 dist-test-slave-03rn krb5kdc[29918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388552, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
W20260926 02:09:13.751593 30614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:09:13.752113 30614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:09:13.752372 30614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:09:13.764302 30614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:09:13.764551 30614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:09:13.764776 30614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:09:13.781669 30614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:43009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.159.148:54103/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:33123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:09:13.783700 30614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:09:13.785705 30614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:09:13.831612 30622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:09:13.831059 30621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:09:13.840916 30614 server_base.cc:1034] running on GCE node
W20260926 02:09:13.841468 30625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:09:13.842505 30614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:09:13.848910 30614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:09:13.850394 30614 hybrid_clock.cc:648] HybridClock initialized: now 1790388553850290 us; error 113 us; skew 500 ppm
I20260926 02:09:13.850878 30614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:09:13 dist-test-slave-03rn krb5kdc[29918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388553, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:09:13.855345 30614 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:09:13.876252 30614 webserver.cc:460] Webserver started at http://127.18.159.130:35079/ using document root <none> and password file <none>
I20260926 02:09:13.877452 30614 fs_manager.cc:359] Metadata directory not provided
I20260926 02:09:13.877701 30614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:09:13.878136 30614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:09:13 dist-test-slave-03rn krb5kdc[29918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388553, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
I20260926 02:09:13.917603 30614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "0ae960777dd142bb830f79001ba54e6e"
format_stamp: "Formatted at 2026-09-26 02:09:13 on dist-test-slave-03rn"
server_key: "8eac3cb6596dfb96cb6ea37c3cb573f3"
server_key_iv: "f10aa40109327faae2508ec93aba5d2d"
server_key_version: "kuduclusterkey@0"
I20260926 02:09:13.918841 30614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "0ae960777dd142bb830f79001ba54e6e"
format_stamp: "Formatted at 2026-09-26 02:09:13 on dist-test-slave-03rn"
server_key: "8eac3cb6596dfb96cb6ea37c3cb573f3"
server_key_iv: "f10aa40109327faae2508ec93aba5d2d"
server_key_version: "kuduclusterkey@0"
I20260926 02:09:13.933243 30614 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.004s sys 0.008s
I20260926 02:09:14.001318 30631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:09:14.003729 30614 fs_manager.cc:725] Time spent opening block manager: real 0.027s user 0.011s sys 0.003s
I20260926 02:09:14.010387 30614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "0ae960777dd142bb830f79001ba54e6e"
format_stamp: "Formatted at 2026-09-26 02:09:13 on dist-test-slave-03rn"
server_key: "8eac3cb6596dfb96cb6ea37c3cb573f3"
server_key_iv: "f10aa40109327faae2508ec93aba5d2d"
server_key_version: "kuduclusterkey@0"
I20260926 02:09:14.010860 30614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:09:14.036487 30614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:09:14.042649 30614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:09:14.042995 30614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:09:14.044225 30614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:09:14.046029 30614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:09:14.046121 30614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:09:14.046285 30614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:09:14.046352 30614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:09:14.096120 30614 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:38973
I20260926 02:09:14.096120 30750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:38973 every 8 connection(s)
I20260926 02:09:14.097539 30614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:09:14.100040 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 30614
I20260926 02:09:14.100238 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:09:14.100785 19070 external_mini_cluster.cc:1428] Setting key a486169c7347d1bce1448956169f59d9
Sep 26 02:09:14 dist-test-slave-03rn krb5kdc[29918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388553, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.159.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.131@KRBTEST.COM" created.
I20260926 02:09:14.134299 30753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:14.100498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:48411 (local address 127.18.159.190:33123)
0926 02:09:14.112288 (+ 11790us) server_negotiation.cc:206] Beginning negotiation
0926 02:09:14.112297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:09:14.113387 (+ 1090us) server_negotiation.cc:407] Connection header received
0926 02:09:14.115846 (+ 2459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:09:14.115855 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:09:14.115996 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:09:14.116115 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:09:14.117891 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.119156 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.121054 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.121776 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.121928 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:14.126353 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:09:14.126363 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:09:14.126371 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:09:14.126425 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:09:14.130016 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:09:14.131058 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:09:14.131063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:09:14.131066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:09:14.131196 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:09:14.131738 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:09:14.131773 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:09:14.131800 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:09:14.132330 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:09:14.132551 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 02:09:14.133439 (+ 888us) server_negotiation.cc:299] Negotiation successful
0926 02:09:14.133722 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11598,"thread_start_us":11464,"threads_started":1}
I20260926 02:09:14.134979 30754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:14.101293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:33123 (local address 127.18.159.130:48411)
0926 02:09:14.112775 (+ 11482us) negotiation.cc:107] Waiting for socket to connect
0926 02:09:14.112853 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0926 02:09:14.115467 (+ 2614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:09:14.116440 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:09:14.116452 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:09:14.117019 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:09:14.117682 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:09:14.117701 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.119367 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.119371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:09:14.120876 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:09:14.120890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.122010 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.122014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:09:14.122224 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:14.123057 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:09:14.123081 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:09:14.126027 (+ 2946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:09:14.130189 (+ 4162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:09:14.130197 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:09:14.130215 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:09:14.130866 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:09:14.131310 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:09:14.131313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:09:14.131316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:09:14.131557 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:09:14.132535 (+ 978us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:09:14.132542 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:09:14.133004 (+ 462us) client_negotiation.cc:769] Sending connection context
0926 02:09:14.133285 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 02:09:14.133708 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11113,"thread_start_us":10876,"threads_started":1}
I20260926 02:09:14.136566 30751 heartbeater.cc:347] Connected to a master server at 127.18.159.190:33123
I20260926 02:09:14.136895 30751 heartbeater.cc:464] Registering TS with master...
I20260926 02:09:14.137655 30751 heartbeater.cc:511] Master 127.18.159.190:33123 requested a full tablet report, sending...
I20260926 02:09:14.139236 30394 ts_manager.cc:194] Registered new tserver with Master: 0ae960777dd142bb830f79001ba54e6e (127.18.159.130:38973)
I20260926 02:09:14.140054 30394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:48411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
I20260926 02:09:14.161863 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.159.148:54103/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:33123
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:43009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:09:14.211612 30608 heartbeater.cc:503] Master 127.18.159.190:33123 was elected leader, sending a full tablet report...
W20260926 02:09:14.529670 30757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:09:14.529964 30757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:09:14.530055 30757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:09:14.537750 30757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:09:14.537896 30757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:09:14.538013 30757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:09:14.548399 30757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:43009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.18.159.148:54103/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:33123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:09:14.549706 30757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:09:14.551052 30757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:09:14.564077 30763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:09:14.564321 30762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:09:14.565883 30765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:09:14.566432 30757 server_base.cc:1034] running on GCE node
I20260926 02:09:14.567018 30757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:09:14.567934 30757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:09:14.569133 30757 hybrid_clock.cc:648] HybridClock initialized: now 1790388554569078 us; error 52 us; skew 500 ppm
I20260926 02:09:14.569458 30757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:09:14 dist-test-slave-03rn krb5kdc[29918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388554, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:09:14.572953 30757 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:09:14.574478 30757 webserver.cc:460] Webserver started at http://127.18.159.131:34063/ using document root <none> and password file <none>
I20260926 02:09:14.575253 30757 fs_manager.cc:359] Metadata directory not provided
I20260926 02:09:14.575383 30757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:09:14.575706 30757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:09:14 dist-test-slave-03rn krb5kdc[29918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388554, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
I20260926 02:09:14.594264 30757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "557d701c7fcf48538b2a1642206736e9"
format_stamp: "Formatted at 2026-09-26 02:09:14 on dist-test-slave-03rn"
server_key: "077ef0b5a549bc822e0bc9b5e843b7b5"
server_key_iv: "a0f06eada5bde0633cfcfcf13f1df57f"
server_key_version: "kuduclusterkey@0"
I20260926 02:09:14.595468 30757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "557d701c7fcf48538b2a1642206736e9"
format_stamp: "Formatted at 2026-09-26 02:09:14 on dist-test-slave-03rn"
server_key: "077ef0b5a549bc822e0bc9b5e843b7b5"
server_key_iv: "a0f06eada5bde0633cfcfcf13f1df57f"
server_key_version: "kuduclusterkey@0"
I20260926 02:09:14.602887 30757 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260926 02:09:14.622572 30772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:09:14.624316 30757 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:09:14.624521 30757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "557d701c7fcf48538b2a1642206736e9"
format_stamp: "Formatted at 2026-09-26 02:09:14 on dist-test-slave-03rn"
server_key: "077ef0b5a549bc822e0bc9b5e843b7b5"
server_key_iv: "a0f06eada5bde0633cfcfcf13f1df57f"
server_key_version: "kuduclusterkey@0"
I20260926 02:09:14.624732 30757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:09:14.641393 30757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:09:14.645244 30757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:09:14.645567 30757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:09:14.646587 30757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:09:14.648451 30757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:09:14.648524 30757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:09:14.648691 30757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:09:14.648741 30757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:09:14.686708 30757 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:40835
I20260926 02:09:14.686745 30884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:40835 every 8 connection(s)
I20260926 02:09:14.688098 30757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:09:14.697513 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 30757
I20260926 02:09:14.697701 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:09:14.698205 19070 external_mini_cluster.cc:1428] Setting key 2d54da9f8f6396a80421e39fc2699d9f
Sep 26 02:09:14 dist-test-slave-03rn krb5kdc[29918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388554, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:09:14.711275 30887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:14.690860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:50489 (local address 127.18.159.190:33123)
0926 02:09:14.691359 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0926 02:09:14.691364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:09:14.692537 (+ 1173us) server_negotiation.cc:407] Connection header received
0926 02:09:14.694441 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:09:14.694446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:09:14.694565 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:09:14.694694 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:09:14.696818 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.697920 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.700362 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.701034 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.701172 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:14.705132 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:09:14.705138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:09:14.705142 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:09:14.705178 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:09:14.707996 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:09:14.708976 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:09:14.708981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:09:14.708984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:09:14.709131 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:09:14.709714 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:09:14.709718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:09:14.709720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:09:14.710006 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:09:14.710139 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 02:09:14.710830 (+ 691us) server_negotiation.cc:299] Negotiation successful
0926 02:09:14.711013 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":258,"threads_started":1}
I20260926 02:09:14.712517 30888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:14.691524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:33123 (local address 127.18.159.131:50489)
0926 02:09:14.692336 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0926 02:09:14.692378 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:09:14.694109 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:09:14.694969 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:09:14.694979 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:09:14.695426 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:09:14.696589 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:09:14.696613 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.698140 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.698145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:09:14.700106 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:09:14.700123 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.701308 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.701317 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:09:14.701672 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:14.702490 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:09:14.702515 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:09:14.704746 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:09:14.708242 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:09:14.708251 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:09:14.708267 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:09:14.708763 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:09:14.709278 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:09:14.709282 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:09:14.709284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:09:14.709555 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:09:14.710132 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:09:14.710139 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:09:14.710480 (+ 341us) client_negotiation.cc:769] Sending connection context
0926 02:09:14.710721 (+ 241us) client_negotiation.cc:240] Negotiation successful
0926 02:09:14.711152 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":308,"threads_started":1}
I20260926 02:09:14.714370 30885 heartbeater.cc:347] Connected to a master server at 127.18.159.190:33123
I20260926 02:09:14.714798 30885 heartbeater.cc:464] Registering TS with master...
I20260926 02:09:14.715631 30885 heartbeater.cc:511] Master 127.18.159.190:33123 requested a full tablet report, sending...
I20260926 02:09:14.717178 30394 ts_manager.cc:194] Registered new tserver with Master: 557d701c7fcf48538b2a1642206736e9 (127.18.159.131:40835)
I20260926 02:09:14.718011 30394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:50489
I20260926 02:09:14.727002 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:09:14.745064 30887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:14.731557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40768 (local address 127.18.159.190:33123)
0926 02:09:14.731736 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 02:09:14.731740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:09:14.732319 (+ 579us) server_negotiation.cc:407] Connection header received
0926 02:09:14.732499 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:09:14.732506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:09:14.732668 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:09:14.732768 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:09:14.734696 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.735434 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.736973 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.737389 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.737490 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:14.739901 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:09:14.739905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:09:14.739909 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:09:14.739944 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:09:14.742737 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:09:14.743399 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:09:14.743403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:09:14.743405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:09:14.743521 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:09:14.743846 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:09:14.743849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:09:14.743851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:09:14.744121 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:09:14.744368 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 02:09:14.744648 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 02:09:14.744815 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:09:14.749198 30394 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:09:14.757079 30394 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:09:14.788066 30899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:14.770748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:38973 (local address 127.0.0.1:33122)
0926 02:09:14.772706 (+ 1958us) negotiation.cc:107] Waiting for socket to connect
0926 02:09:14.772746 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 02:09:14.772953 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:09:14.774227 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:09:14.774239 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:09:14.774285 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:09:14.774913 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:09:14.774931 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.778348 (+ 3417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.778358 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:09:14.782110 (+ 3752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:09:14.782141 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.786398 (+ 4257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.786410 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:09:14.786809 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:14.786911 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 02:09:14.787104 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 02:09:14.787405 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1714,"thread_start_us":15385,"threads_started":1}
I20260926 02:09:14.789402 30900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:14.770915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33122 (local address 127.18.159.130:38973)
0926 02:09:14.772048 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0926 02:09:14.772063 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:09:14.773161 (+ 1098us) server_negotiation.cc:407] Connection header received
0926 02:09:14.773272 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:09:14.773283 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:09:14.773594 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:09:14.773838 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:09:14.775182 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.778088 (+ 2906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.782432 (+ 4344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.784862 (+ 2430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.785116 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:14.785397 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0926 02:09:14.788620 (+ 3223us) server_negotiation.cc:299] Negotiation successful
0926 02:09:14.788921 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":1072,"threads_started":1}
I20260926 02:09:14.794979 30686 tablet_service.cc:1467] Processing CreateTablet for tablet 1112d952d567489ca7b2b4b86e3a2793 (DEFAULT_TABLE table=test-table [id=b2f030af1d5d4e6bb4b396bebdc9b62d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:09:14.796789 30686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1112d952d567489ca7b2b4b86e3a2793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:09:14.797456 30898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:14.771048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:40835 (local address 127.0.0.1:50008)
0926 02:09:14.786465 (+ 15417us) negotiation.cc:107] Waiting for socket to connect
0926 02:09:14.786494 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 02:09:14.786711 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:09:14.788174 (+ 1463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:09:14.788184 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:09:14.788226 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:09:14.788691 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:09:14.788707 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.791003 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.791008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:09:14.793617 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:09:14.793630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.796123 (+ 2493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.796129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:09:14.796776 (+ 647us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:14.796816 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 02:09:14.796908 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 02:09:14.797066 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15267,"thread_start_us":1258,"threads_started":1}
I20260926 02:09:14.797636 30901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:14.771754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50008 (local address 127.18.159.131:40835)
0926 02:09:14.786127 (+ 14373us) server_negotiation.cc:206] Beginning negotiation
0926 02:09:14.786142 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:09:14.786928 (+ 786us) server_negotiation.cc:407] Connection header received
0926 02:09:14.787083 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:09:14.787096 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:09:14.787479 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:09:14.787745 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:09:14.788983 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.790833 (+ 1850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.793810 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.795546 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.795758 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:14.795950 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 02:09:14.797092 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0926 02:09:14.797270 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14105,"thread_start_us":13877,"threads_started":1}
I20260926 02:09:14.799597 30903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:14.773863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.18.159.129:37469)
0926 02:09:14.790686 (+ 16823us) server_negotiation.cc:206] Beginning negotiation
0926 02:09:14.790695 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:09:14.790725 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:09:14.790823 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:09:14.790832 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:09:14.791129 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:09:14.792047 (+ 918us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:09:14.792323 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.793839 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.796246 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.798409 (+ 2163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.798606 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:14.798789 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 02:09:14.799035 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 02:09:14.799243 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16632,"thread_start_us":17409,"threads_started":1}
I20260926 02:09:14.799800 30899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:14.773194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:37469 (local address 127.0.0.1:34204)
0926 02:09:14.790244 (+ 17050us) negotiation.cc:107] Waiting for socket to connect
0926 02:09:14.790271 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:09:14.790407 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:09:14.791594 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:09:14.791598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:09:14.791625 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:09:14.791861 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:09:14.791871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.794009 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.794015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:09:14.796004 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:09:14.796020 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.798592 (+ 2572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.798598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:09:14.798804 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:14.798856 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 02:09:14.798962 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 02:09:14.799511 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16896,"thread_start_us":19197,"threads_started":1}
I20260926 02:09:14.802084 30820 tablet_service.cc:1467] Processing CreateTablet for tablet 1112d952d567489ca7b2b4b86e3a2793 (DEFAULT_TABLE table=test-table [id=b2f030af1d5d4e6bb4b396bebdc9b62d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:09:14.803798 30820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1112d952d567489ca7b2b4b86e3a2793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:09:14.804491 30543 tablet_service.cc:1467] Processing CreateTablet for tablet 1112d952d567489ca7b2b4b86e3a2793 (DEFAULT_TABLE table=test-table [id=b2f030af1d5d4e6bb4b396bebdc9b62d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:09:14.806173 30543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1112d952d567489ca7b2b4b86e3a2793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:09:14.811705 30904 tablet_bootstrap.cc:492] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e: Bootstrap starting.
I20260926 02:09:14.815644 30904 tablet_bootstrap.cc:654] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e: Neither blocks nor log segments found. Creating new log.
I20260926 02:09:14.817679 30904 log.cc:824] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e: Log is configured to *not* fsync() on all Append() calls
I20260926 02:09:14.820675 30904 tablet_bootstrap.cc:492] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e: No bootstrap required, opened a new log
I20260926 02:09:14.820773 30906 tablet_bootstrap.cc:492] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9: Bootstrap starting.
I20260926 02:09:14.821130 30904 ts_tablet_manager.cc:1397] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260926 02:09:14.822449 30907 tablet_bootstrap.cc:492] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b: Bootstrap starting.
I20260926 02:09:14.824476 30906 tablet_bootstrap.cc:654] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9: Neither blocks nor log segments found. Creating new log.
I20260926 02:09:14.825807 30906 log.cc:824] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9: Log is configured to *not* fsync() on all Append() calls
I20260926 02:09:14.827363 30907 tablet_bootstrap.cc:654] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b: Neither blocks nor log segments found. Creating new log.
I20260926 02:09:14.828948 30907 log.cc:824] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b: Log is configured to *not* fsync() on all Append() calls
I20260926 02:09:14.830924 30906 tablet_bootstrap.cc:492] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9: No bootstrap required, opened a new log
I20260926 02:09:14.831256 30906 ts_tablet_manager.cc:1397] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260926 02:09:14.832337 30907 tablet_bootstrap.cc:492] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b: No bootstrap required, opened a new log
I20260926 02:09:14.832656 30907 ts_tablet_manager.cc:1397] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260926 02:09:14.834950 30904 raft_consensus.cc:348] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae960777dd142bb830f79001ba54e6e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38973 } } peers { permanent_uuid: "557d701c7fcf48538b2a1642206736e9" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 40835 } } peers { permanent_uuid: "c6ed22dcbf9f41ab90fbc26e6cec480b" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 37469 } }
I20260926 02:09:14.835433 30904 raft_consensus.cc:374] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:09:14.835588 30904 raft_consensus.cc:729] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ae960777dd142bb830f79001ba54e6e, State: Initialized, Role: FOLLOWER
I20260926 02:09:14.836308 30904 consensus_queue.cc:260] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae960777dd142bb830f79001ba54e6e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38973 } } peers { permanent_uuid: "557d701c7fcf48538b2a1642206736e9" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 40835 } } peers { permanent_uuid: "c6ed22dcbf9f41ab90fbc26e6cec480b" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 37469 } }
I20260926 02:09:14.837568 30751 heartbeater.cc:503] Master 127.18.159.190:33123 was elected leader, sending a full tablet report...
I20260926 02:09:14.839790 30904 ts_tablet_manager.cc:1428] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e: Time spent starting tablet: real 0.018s user 0.014s sys 0.000s
I20260926 02:09:14.843102 30906 raft_consensus.cc:348] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae960777dd142bb830f79001ba54e6e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38973 } } peers { permanent_uuid: "557d701c7fcf48538b2a1642206736e9" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 40835 } } peers { permanent_uuid: "c6ed22dcbf9f41ab90fbc26e6cec480b" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 37469 } }
I20260926 02:09:14.843612 30906 raft_consensus.cc:374] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:09:14.843744 30906 raft_consensus.cc:729] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 557d701c7fcf48538b2a1642206736e9, State: Initialized, Role: FOLLOWER
I20260926 02:09:14.843989 30907 raft_consensus.cc:348] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae960777dd142bb830f79001ba54e6e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38973 } } peers { permanent_uuid: "557d701c7fcf48538b2a1642206736e9" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 40835 } } peers { permanent_uuid: "c6ed22dcbf9f41ab90fbc26e6cec480b" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 37469 } }
I20260926 02:09:14.844482 30907 raft_consensus.cc:374] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:09:14.844606 30907 raft_consensus.cc:729] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ed22dcbf9f41ab90fbc26e6cec480b, State: Initialized, Role: FOLLOWER
I20260926 02:09:14.844446 30906 consensus_queue.cc:260] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae960777dd142bb830f79001ba54e6e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38973 } } peers { permanent_uuid: "557d701c7fcf48538b2a1642206736e9" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 40835 } } peers { permanent_uuid: "c6ed22dcbf9f41ab90fbc26e6cec480b" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 37469 } }
I20260926 02:09:14.845355 30907 consensus_queue.cc:260] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae960777dd142bb830f79001ba54e6e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38973 } } peers { permanent_uuid: "557d701c7fcf48538b2a1642206736e9" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 40835 } } peers { permanent_uuid: "c6ed22dcbf9f41ab90fbc26e6cec480b" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 37469 } }
I20260926 02:09:14.847404 30907 ts_tablet_manager.cc:1428] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260926 02:09:14.848853 30885 heartbeater.cc:503] Master 127.18.159.190:33123 was elected leader, sending a full tablet report...
I20260926 02:09:14.849666 30906 ts_tablet_manager.cc:1428] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9: Time spent starting tablet: real 0.018s user 0.010s sys 0.007s
W20260926 02:09:14.849768 30752 tablet.cc:2369] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:09:14.917687 30609 tablet.cc:2369] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:09:14.919764 30912 raft_consensus.cc:482] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:09:14.920073 30912 raft_consensus.cc:504] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae960777dd142bb830f79001ba54e6e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38973 } } peers { permanent_uuid: "557d701c7fcf48538b2a1642206736e9" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 40835 } } peers { permanent_uuid: "c6ed22dcbf9f41ab90fbc26e6cec480b" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 37469 } }
I20260926 02:09:14.921787 30912 leader_election.cc:288] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ae960777dd142bb830f79001ba54e6e (127.18.159.130:38973), 557d701c7fcf48538b2a1642206736e9 (127.18.159.131:40835)
I20260926 02:09:14.931854 30901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:14.922303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:37459 (local address 127.18.159.131:40835)
0926 02:09:14.922517 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:09:14.922523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:09:14.923037 (+ 514us) server_negotiation.cc:407] Connection header received
0926 02:09:14.923285 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:09:14.923289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:09:14.923387 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:09:14.923480 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:09:14.924695 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.925897 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.929103 (+ 3206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.930435 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.930607 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:14.930683 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 02:09:14.931438 (+ 755us) server_negotiation.cc:299] Negotiation successful
0926 02:09:14.931574 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 02:09:14.931915 30900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:14.922087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:57801 (local address 127.18.159.130:38973)
0926 02:09:14.922352 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 02:09:14.922361 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:09:14.923376 (+ 1015us) server_negotiation.cc:407] Connection header received
0926 02:09:14.923488 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:09:14.923493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:09:14.923563 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:09:14.923650 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:09:14.924785 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.925907 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.929138 (+ 3231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.930435 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.930607 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:14.930747 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 02:09:14.931423 (+ 676us) server_negotiation.cc:299] Negotiation successful
0926 02:09:14.931600 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 02:09:14.932042 30914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:14.922458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:38973 (local address 127.18.159.129:57801)
0926 02:09:14.923265 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0926 02:09:14.923280 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:09:14.923390 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:09:14.923685 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:09:14.923690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:09:14.923730 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:09:14.924615 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:09:14.924629 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.926052 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.926057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:09:14.928886 (+ 2829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:09:14.928906 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.930799 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.930804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:09:14.931113 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:14.931246 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 02:09:14.931356 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 02:09:14.931517 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":217,"thread_start_us":480,"threads_started":1}
I20260926 02:09:14.932063 30913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:14.922061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:40835 (local address 127.18.159.129:37459)
0926 02:09:14.922914 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0926 02:09:14.922931 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:09:14.923128 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:09:14.923538 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:09:14.923549 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:09:14.923570 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:09:14.924510 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:09:14.924529 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.926051 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.926057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:09:14.928886 (+ 2829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:09:14.928906 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.930788 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.930797 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:09:14.931114 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:14.931260 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 02:09:14.931406 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 02:09:14.931519 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":379,"thread_start_us":518,"threads_started":1}
I20260926 02:09:14.933320 30840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1112d952d567489ca7b2b4b86e3a2793" candidate_uuid: "c6ed22dcbf9f41ab90fbc26e6cec480b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "557d701c7fcf48538b2a1642206736e9" is_pre_election: true
I20260926 02:09:14.933323 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1112d952d567489ca7b2b4b86e3a2793" candidate_uuid: "c6ed22dcbf9f41ab90fbc26e6cec480b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae960777dd142bb830f79001ba54e6e" is_pre_election: true
I20260926 02:09:14.933852 30840 raft_consensus.cc:2454] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ed22dcbf9f41ab90fbc26e6cec480b in term 0.
I20260926 02:09:14.933887 30706 raft_consensus.cc:2454] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ed22dcbf9f41ab90fbc26e6cec480b in term 0.
I20260926 02:09:14.934633 30497 leader_election.cc:302] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ae960777dd142bb830f79001ba54e6e, 557d701c7fcf48538b2a1642206736e9, c6ed22dcbf9f41ab90fbc26e6cec480b; no voters:
I20260926 02:09:14.935273 30912 raft_consensus.cc:2781] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:09:14.935490 30912 raft_consensus.cc:482] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:09:14.935631 30912 raft_consensus.cc:3037] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:09:14.938030 30912 raft_consensus.cc:504] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae960777dd142bb830f79001ba54e6e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38973 } } peers { permanent_uuid: "557d701c7fcf48538b2a1642206736e9" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 40835 } } peers { permanent_uuid: "c6ed22dcbf9f41ab90fbc26e6cec480b" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 37469 } }
I20260926 02:09:14.938720 30912 leader_election.cc:288] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b [CANDIDATE]: Term 1 election: Requested vote from peers 0ae960777dd142bb830f79001ba54e6e (127.18.159.130:38973), 557d701c7fcf48538b2a1642206736e9 (127.18.159.131:40835)
I20260926 02:09:14.939335 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1112d952d567489ca7b2b4b86e3a2793" candidate_uuid: "c6ed22dcbf9f41ab90fbc26e6cec480b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae960777dd142bb830f79001ba54e6e"
I20260926 02:09:14.939329 30840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1112d952d567489ca7b2b4b86e3a2793" candidate_uuid: "c6ed22dcbf9f41ab90fbc26e6cec480b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "557d701c7fcf48538b2a1642206736e9"
I20260926 02:09:14.939558 30706 raft_consensus.cc:3037] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:09:14.939618 30840 raft_consensus.cc:3037] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9 [term 0 FOLLOWER]: Advancing to term 1
W20260926 02:09:14.939883 30886 tablet.cc:2369] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:09:14.942432 30840 raft_consensus.cc:2454] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ed22dcbf9f41ab90fbc26e6cec480b in term 1.
I20260926 02:09:14.942552 30706 raft_consensus.cc:2454] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ed22dcbf9f41ab90fbc26e6cec480b in term 1.
I20260926 02:09:14.942987 30497 leader_election.cc:302] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 557d701c7fcf48538b2a1642206736e9, c6ed22dcbf9f41ab90fbc26e6cec480b; no voters:
I20260926 02:09:14.943555 30912 raft_consensus.cc:2781] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:09:14.944087 30912 raft_consensus.cc:686] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b [term 1 LEADER]: Becoming Leader. State: Replica: c6ed22dcbf9f41ab90fbc26e6cec480b, State: Running, Role: LEADER
I20260926 02:09:14.944635 30912 consensus_queue.cc:237] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae960777dd142bb830f79001ba54e6e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38973 } } peers { permanent_uuid: "557d701c7fcf48538b2a1642206736e9" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 40835 } } peers { permanent_uuid: "c6ed22dcbf9f41ab90fbc26e6cec480b" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 37469 } }
I20260926 02:09:14.949361 30394 catalog_manager.cc:5495] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ed22dcbf9f41ab90fbc26e6cec480b (127.18.159.129). New cstate: current_term: 1 leader_uuid: "c6ed22dcbf9f41ab90fbc26e6cec480b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae960777dd142bb830f79001ba54e6e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "557d701c7fcf48538b2a1642206736e9" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 40835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ed22dcbf9f41ab90fbc26e6cec480b" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 37469 } health_report { overall_health: HEALTHY } } }
I20260926 02:09:14.999570 30903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:14.993476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34220 (local address 127.18.159.129:37469)
0926 02:09:14.993686 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 02:09:14.993692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:09:14.993712 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:09:14.993932 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:09:14.993936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:09:14.994029 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:09:14.994188 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:09:14.994611 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.995354 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.996865 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:14.997681 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:14.997837 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:14.998371 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:09:14.998912 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:09:14.999081 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 02:09:14.999184 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 02:09:14.999291 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:09:15.008817 30706 raft_consensus.cc:1261] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e [term 1 FOLLOWER]: Refusing update from remote peer c6ed22dcbf9f41ab90fbc26e6cec480b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:09:15.008817 30840 raft_consensus.cc:1261] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9 [term 1 FOLLOWER]: Refusing update from remote peer c6ed22dcbf9f41ab90fbc26e6cec480b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:09:15.010108 30915 consensus_queue.cc:1035] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b [LEADER]: Connected to new peer: Peer: permanent_uuid: "557d701c7fcf48538b2a1642206736e9" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 40835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:09:15.010569 30912 consensus_queue.cc:1035] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ae960777dd142bb830f79001ba54e6e" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:09:15.026050 30918 mvcc.cc:205] Tried to move back new op lower bound from 7333431521304596480 to 7333431521057222656. Current Snapshot: MvccSnapshot[applied={T|T < 7333431521304596480}]
I20260926 02:09:15.036286 30919 mvcc.cc:205] Tried to move back new op lower bound from 7333431521304596480 to 7333431521057222656. Current Snapshot: MvccSnapshot[applied={T|T < 7333431521304596480}]
I20260926 02:09:15.042693 30394 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:09:15.042997 30394 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:09:15.047066 30394 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28d81752911f44d0be55457aecca5ac2: Sending DeleteTablet for 3 replicas of tablet 1112d952d567489ca7b2b4b86e3a2793
I20260926 02:09:15.048557 30686 tablet_service.cc:1514] Processing DeleteTablet for tablet 1112d952d567489ca7b2b4b86e3a2793 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:09:15 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:33122
I20260926 02:09:15.048856 30820 tablet_service.cc:1514] Processing DeleteTablet for tablet 1112d952d567489ca7b2b4b86e3a2793 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:09:15 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:50008
I20260926 02:09:15.049463 30543 tablet_service.cc:1514] Processing DeleteTablet for tablet 1112d952d567489ca7b2b4b86e3a2793 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:09:15 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:34204
I20260926 02:09:15.050318 30929 tablet_replica.cc:330] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9: stopping tablet replica
I20260926 02:09:15.050807 30929 raft_consensus.cc:2229] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:09:15.051254 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 30475
I20260926 02:09:15.051364 30929 raft_consensus.cc:2258] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:09:15.052227 30928 tablet_replica.cc:330] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e: stopping tablet replica
I20260926 02:09:15.052699 30928 raft_consensus.cc:2229] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:09:15.053225 30928 raft_consensus.cc:2258] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:09:15.055706 30928 ts_tablet_manager.cc:1905] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:09:15.060602 30929 ts_tablet_manager.cc:1905] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:09:15.065270 30928 ts_tablet_manager.cc:1918] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:09:15.065557 30928 log.cc:1196] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/1112d952d567489ca7b2b4b86e3a2793
I20260926 02:09:15.066174 30928 ts_tablet_manager.cc:1939] T 1112d952d567489ca7b2b4b86e3a2793 P 0ae960777dd142bb830f79001ba54e6e: Deleting consensus metadata
I20260926 02:09:15.066203 30930 tablet_replica.cc:330] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b: stopping tablet replica
I20260926 02:09:15.066874 30930 raft_consensus.cc:2229] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:09:15.068100 30930 raft_consensus.cc:2258] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:09:15.068285 30378 catalog_manager.cc:4841] TS 0ae960777dd142bb830f79001ba54e6e (127.18.159.130:38973): tablet 1112d952d567489ca7b2b4b86e3a2793 (table test-table [id=b2f030af1d5d4e6bb4b396bebdc9b62d]) successfully deleted
I20260926 02:09:15.073086 30930 ts_tablet_manager.cc:1905] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:09:15.073746 30929 ts_tablet_manager.cc:1918] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:09:15.073976 30929 log.cc:1196] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/1112d952d567489ca7b2b4b86e3a2793
I20260926 02:09:15.074553 30929 ts_tablet_manager.cc:1939] T 1112d952d567489ca7b2b4b86e3a2793 P 557d701c7fcf48538b2a1642206736e9: Deleting consensus metadata
Sep 26 02:09:15 dist-test-slave-03rn krb5kdc[29918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:09:15.076627 30379 catalog_manager.cc:4841] TS 557d701c7fcf48538b2a1642206736e9 (127.18.159.131:40835): tablet 1112d952d567489ca7b2b4b86e3a2793 (table test-table [id=b2f030af1d5d4e6bb4b396bebdc9b62d]) successfully deleted
I20260926 02:09:15.081528 30930 ts_tablet_manager.cc:1918] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:09:15.081745 30930 log.cc:1196] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/1112d952d567489ca7b2b4b86e3a2793
I20260926 02:09:15.082851 30930 ts_tablet_manager.cc:1939] T 1112d952d567489ca7b2b4b86e3a2793 P c6ed22dcbf9f41ab90fbc26e6cec480b: Deleting consensus metadata
I20260926 02:09:15.084582 30379 catalog_manager.cc:4841] TS c6ed22dcbf9f41ab90fbc26e6cec480b (127.18.159.129:37469): tablet 1112d952d567489ca7b2b4b86e3a2793 (table test-table [id=b2f030af1d5d4e6bb4b396bebdc9b62d]) successfully deleted
I20260926 02:09:15.085238 30903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:15.054420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34226 (local address 127.18.159.129:37469)
0926 02:09:15.054617 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 02:09:15.054624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:09:15.063592 (+ 8968us) server_negotiation.cc:407] Connection header received
0926 02:09:15.064113 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:09:15.064119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:09:15.064242 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:09:15.064318 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:09:15.066599 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:15.067882 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:15.069704 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:15.070188 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:15.070364 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:15.076228 (+ 5864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:09:15.076247 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:09:15.076268 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:09:15.076321 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:09:15.080901 (+ 4580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:09:15.081916 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:09:15.081933 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:09:15.081938 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:09:15.082141 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:09:15.083037 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:09:15.083043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:09:15.083047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:09:15.083585 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:09:15.083893 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 02:09:15.084601 (+ 708us) server_negotiation.cc:299] Negotiation successful
0926 02:09:15.084867 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 02:09:15.086676 30603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:09:15.202569 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 30475
I20260926 02:09:15.220990 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 30614
Sep 26 02:09:15 dist-test-slave-03rn krb5kdc[29918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.130@KRBTEST.COM
I20260926 02:09:15.237540 30900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:15.221890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33136 (local address 127.18.159.130:38973)
0926 02:09:15.222077 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 02:09:15.222082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:09:15.222100 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:09:15.222231 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:09:15.222235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:09:15.222296 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:09:15.222380 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:09:15.223691 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:15.224682 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:15.226088 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:15.226490 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:15.226585 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:15.231006 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:09:15.231032 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:09:15.231049 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:09:15.231092 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:09:15.234671 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:09:15.235506 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:09:15.235514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:09:15.235516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:09:15.235666 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:09:15.236039 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:09:15.236043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:09:15.236045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:09:15.236550 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:09:15.236784 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 02:09:15.237100 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 02:09:15.237265 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 02:09:15.238307 30746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:09:15.329061 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 30614
I20260926 02:09:15.346062 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 30757
Sep 26 02:09:15 dist-test-slave-03rn krb5kdc[29918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.131@KRBTEST.COM
I20260926 02:09:15.361806 30901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:09:15.346872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50022 (local address 127.18.159.131:40835)
0926 02:09:15.347053 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 02:09:15.347057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:09:15.347200 (+ 143us) server_negotiation.cc:407] Connection header received
0926 02:09:15.347411 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:09:15.347417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:09:15.347506 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:09:15.347607 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:09:15.349140 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:15.349856 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:15.351181 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:09:15.351668 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:09:15.351795 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:09:15.356339 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:09:15.356355 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:09:15.356372 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:09:15.356412 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:09:15.359097 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:09:15.359830 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:09:15.359853 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:09:15.359855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:09:15.359978 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:09:15.360407 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:09:15.360410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:09:15.360412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:09:15.360769 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:09:15.361007 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 02:09:15.361278 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 02:09:15.361439 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:09:15.362515 30880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:09:15.451552 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 30757
I20260926 02:09:15.470726 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 30362
I20260926 02:09:15.472564 30424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:09:15.599789 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 30362
I20260926 02:09:15.622605 19070 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 02:09:15.959007 19070 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 02:09:15.959277 19070 mini_ranger.cc:66] Stopping Ranger...
I20260926 02:09:16.305718 19070 mini_ranger.cc:68] Stopped Ranger
2026-09-26 02:09:16.306 UTC [29948] LOG: received smart shutdown request
2026-09-26 02:09:16.310 UTC [29948] LOG: background worker "logical replication launcher" (PID 29957) exited with exit code 1
2026-09-26 02:09:16.310 UTC [29952] LOG: shutting down
2026-09-26 02:09:16.494 UTC [29948] LOG: database system is shut down
2026-09-26T02:09:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52583 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:09:16 dist-test-slave-03rn krb5kdc[30944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:09:16 dist-test-slave-03rn krb5kdc[30944](info): set up 2 sockets
Sep 26 02:09:16 dist-test-slave-03rn krb5kdc[30944](info): commencing operation
krb5kdc: starting...
W20260926 02:09:17.773319 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:09:17 dist-test-slave-03rn krb5kdc[30944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:09:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:09:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.159.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.159.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.159.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/rangeradmin_127.18.159.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.159.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/rangeradmin_127.18.159.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.159.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.159.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.159.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/rangerlookup_127.18.159.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.159.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/rangerlookup_127.18.159.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.159.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.159.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.159.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/HTTP_127.18.159.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.159.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/HTTP_127.18.159.148@KRBTEST.COM.keytab.
I20260926 02:09:18.115773 19070 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8FmYUC/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/postgres -l logfile start
2026-09-26 02:09:20.714 UTC [30974] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 02:09:20.714 UTC [30974] LOG: listening on IPv4 address "127.18.159.148", port 50593
2026-09-26 02:09:20.717 UTC [30974] LOG: listening on Unix socket "/tmp/.s.PGSQL.50593"
2026-09-26 02:09:20.742 UTC [30977] LOG: database system was shut down at 2026-09-26 02:09:19 UTC
2026-09-26 02:09:20.747 UTC [30974] LOG: database system is ready to accept connections
I20260926 02:09:21.721210 19070 mini_postgres.cc:96] Postgres bound to 50593
2026-09-26 02:09:21.739 UTC [30985] FATAL: database "slave" does not exist
127.18.159.148:50593 - accepting connections
I20260926 02:09:21.740381 19070 mini_ranger.cc:161] Starting Ranger...
I20260926 02:09:21.768725 19070 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 02:09:22.291117 19070 mini_ranger.cc:87] Created ranger Postgres database
I20260926 02:09:22.291217 19070 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ranger-admin
2026-09-26 02:09:22,602 [I] DB FLAVOR :POSTGRES
2026-09-26 02:09:22,603 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 02:09:22,603 [I] Checking connection..
2026-09-26 02:09:22,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 02:09:22,859 [I] Checking connection passed.
2026-09-26 02:09:22,859 [I] --------- Verifying version history table ---------
2026-09-26 02:09:22,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:09:23,130 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 02:09:23,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:09:23,375 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 02:09:23,376 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 02:09:23,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 02:09:23.649 UTC [31049] WARNING: there is no transaction in progress
2026-09-26 02:09:23,656 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 02:09:23,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:09:23,913 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 02:09:23,913 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 02:09:23,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 02:09:24,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 02:09:24,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-03rn.c.gcp-upstream.internal','N') ;"
2026-09-26 02:09:24,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 02:09:24,940 [I] Table x_portal_user does not exist in database ranger
2026-09-26 02:09:24,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 02:09:25,213 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 02:09:25,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 02:09:25,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 02:09:25,725 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 02:09:25,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 02:09:26.751 UTC [31157] WARNING: there is no transaction in progress
2026-09-26 02:09:26.768 UTC [31157] WARNING: there is no transaction in progress
2026-09-26 02:09:26.786 UTC [31157] WARNING: there is no transaction in progress
2026-09-26 02:09:26.803 UTC [31157] WARNING: there is no transaction in progress
2026-09-26 02:09:26.824 UTC [31157] WARNING: there is no transaction in progress
2026-09-26 02:09:26.844 UTC [31157] WARNING: there is no transaction in progress
2026-09-26 02:09:26.904 UTC [31157] WARNING: there is no transaction in progress
2026-09-26 02:09:26.912 UTC [31157] WARNING: there is no transaction in progress
2026-09-26 02:09:26.927 UTC [31157] WARNING: there is no transaction in progress
2026-09-26 02:09:26.939 UTC [31157] WARNING: there is no transaction in progress
2026-09-26 02:09:26.951 UTC [31157] WARNING: there is no transaction in progress
2026-09-26 02:09:26.966 UTC [31157] WARNING: there is no transaction in progress
2026-09-26 02:09:26.977 UTC [31157] WARNING: there is no transaction in progress
2026-09-26 02:09:26.985 UTC [31157] WARNING: there is no transaction in progress
2026-09-26 02:09:26.992 UTC [31157] WARNING: there is no transaction in progress
2026-09-26 02:09:27.000 UTC [31157] WARNING: there is no transaction in progress
2026-09-26 02:09:27,730 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 02:09:27,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 02:09:27,978 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 02:09:27,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 02:09:28,227 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 02:09:28,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 02:09:28,480 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 02:09:28,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 02:09:28,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 02:09:28,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-03rn.c.gcp-upstream.internal';"
2026-09-26 02:09:29,229 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 02:09:29,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 02:09:29,470 [I] DB_PATCHES have already been applied
I20260926 02:09:29.476830 19070 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ranger-admin:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8FmYUC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 02:09:29.476923 19070 mini_ranger.cc:193] Using host: 127.18.159.148
I20260926 02:09:29.492362 19070 mini_ranger.cc:227] Ranger admin URL: http://127.18.159.148:47781
Sep 26, 2026 2:09:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:09:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:09:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 2:09:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 2:09:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 2:09:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 2:09:30 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 02:09:30.649662 19070 mini_ranger.cc:160] Time spent starting Ranger: real 8.909s user 0.007s sys 0.035s
Sep 26, 2026 2:09:51 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 2:09:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 2:09:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 2:09:52 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 2:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 2:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 2:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 2:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 2:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 2:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 2:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 2:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 2:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 2:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 2:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 2:09:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 2:09:52 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 2:09:52 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 02:09:53.614244 19070 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.159.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.159.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.159.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/rangerkms_127.18.159.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.159.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/rangerkms_127.18.159.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.159.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.159.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.159.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/HTTP_127.18.159.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.159.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/HTTP_127.18.159.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 02:09:53 dist-test-slave-03rn krb5kdc[30944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388593, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 02:09:53.785949 19070 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 02:09:53.807618 19070 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 02:09:54.706035 19070 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 02:09:54.706154 19070 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ranger-kms
I20260926 02:09:54.706200 19070 mini_ranger_kms.cc:227] Using postgres at 127.18.159.148:50593
2026-09-26 02:09:55,076 [I] DB FLAVOR :POSTGRES
2026-09-26 02:09:55,077 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 02:09:55,077 [I] Checking connection
2026-09-26 02:09:55,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 02:09:55,316 [I] connection success
2026-09-26 02:09:55,316 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 02:09:55,316 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 02:09:55,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 02:09:55,584 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 02:09:55,584 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 02:09:55,584 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 02:09:55,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.159.148:50593/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 02:09:55,892 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 02:09:56.447889 19070 mini_ranger_kms.cc:310] Created kms service
I20260926 02:09:56.665237 19070 mini_ranger_kms.cc:326] Created kudu user
I20260926 02:09:56.713542 19070 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 02:09:56 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388593, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
I20260926 02:09:57.119119 19070 mini_ranger_kms.cc:384] Added ranger policy
I20260926 02:09:57.119453 19070 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ranger-kms:/tmp/dist-test-task8FmYUC/build/asan/bin/postgresql.jar:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8FmYUC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 02:09:57.119511 19070 mini_ranger_kms.cc:240] Using host: 127.18.159.148
I20260926 02:09:57.136734 19070 mini_ranger_kms.cc:276] Ranger KMS PID: 31338
I20260926 02:09:57.136878 19070 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.159.148:37729
Sep 26, 2026 2:09:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:09:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:09:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 2:09:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 2:09:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 2:09:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 02:09:59 dist-test-slave-03rn krb5kdc[30944](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790388599, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.159.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:09:59.607576 19070 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.822s user 0.031s sys 0.354s
I20260926 02:09:59.607806 19070 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 02:09:59.608007 19070 mini_ranger_kms.cc:401] 127.18.159.148:37729/kms/v1/keys
Sep 26 02:09:59 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388599, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.159.148@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
Sep 26 02:10:00 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388599, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.159.148@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
Sep 26 02:10:02 dist-test-slave-03rn krb5kdc[30944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:10:02.732354 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:45111
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:45111
--ranger_config_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.159.148:37729/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:10:03.072966 31388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:03.073290 31388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:03.073490 31388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:03.081234 31388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:10:03.081363 31388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:03.081408 31388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:03.081498 31388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:10:03.081573 31388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:10:03.092182 31388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.159.148:37729/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:45111
--ranger_config_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:45111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:03.093586 31388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:03.095134 31388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:03.108036 31393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:03.108119 31396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:03.108175 31394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:03.108210 31388 server_base.cc:1034] running on GCE node
I20260926 02:10:03.109619 31388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:03.111160 31388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:03.112371 31388 hybrid_clock.cc:648] HybridClock initialized: now 1790388603112330 us; error 39 us; skew 500 ppm
I20260926 02:10:03.112831 31388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:03 dist-test-slave-03rn krb5kdc[30944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388603, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:03.121503 31388 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:10:03.123193 31388 webserver.cc:460] Webserver started at http://127.18.159.190:35621/ using document root <none> and password file <none>
I20260926 02:10:03.123883 31388 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:03.123991 31388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:03.124346 31388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:10:03 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388603, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
I20260926 02:10:03.175570 31388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "c5d2f8f2cdf8421098dac41087354e7f"
format_stamp: "Formatted at 2026-09-26 02:10:03 on dist-test-slave-03rn"
server_key: "807510ca7d4febb3c8a3d551dd591288"
server_key_iv: "2187cfadbadcc5b253147bdcef6ee688"
server_key_version: "kuduclusterkey@0"
I20260926 02:10:03.176723 31388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "c5d2f8f2cdf8421098dac41087354e7f"
format_stamp: "Formatted at 2026-09-26 02:10:03 on dist-test-slave-03rn"
server_key: "807510ca7d4febb3c8a3d551dd591288"
server_key_iv: "2187cfadbadcc5b253147bdcef6ee688"
server_key_version: "kuduclusterkey@0"
I20260926 02:10:03.184001 31388 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260926 02:10:03.201110 31403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:03.202804 31388 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:10:03.203015 31388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "c5d2f8f2cdf8421098dac41087354e7f"
format_stamp: "Formatted at 2026-09-26 02:10:03 on dist-test-slave-03rn"
server_key: "807510ca7d4febb3c8a3d551dd591288"
server_key_iv: "2187cfadbadcc5b253147bdcef6ee688"
server_key_version: "kuduclusterkey@0"
I20260926 02:10:03.203246 31388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:03.215570 31388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:03.219800 31388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:03.220214 31388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:03.239686 31388 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:45111
I20260926 02:10:03.239694 31454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:45111 every 8 connection(s)
I20260926 02:10:03.241228 31388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:10:03.245182 31455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:10:03.247874 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 31388
I20260926 02:10:03.248106 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:10:03.248953 19070 external_mini_cluster.cc:1428] Setting key aa5f3ae05765c199e289ff7bf77338a2
I20260926 02:10:03.259625 31455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f: Bootstrap starting.
I20260926 02:10:03.263829 31455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f: Neither blocks nor log segments found. Creating new log.
Sep 26 02:10:03 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:10:03.265224 31455 log.cc:824] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f: Log is configured to *not* fsync() on all Append() calls
I20260926 02:10:03.269320 31455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f: No bootstrap required, opened a new log
I20260926 02:10:03.275913 31458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:03.251263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.18.159.190:45111)
0926 02:10:03.252222 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:03.252246 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:03.252291 (+ 45us) server_negotiation.cc:407] Connection header received
0926 02:10:03.254213 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:03.254254 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:03.254899 (+ 645us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:03.255341 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:03.257114 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:03.258551 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:03.260564 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:03.261319 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:03.261530 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:03.266609 (+ 5079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:03.266618 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:03.266637 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:03.266679 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:03.270650 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:03.271549 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:03.271564 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:03.271575 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:03.271800 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:03.272344 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:03.272350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:03.272354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:03.273157 (+ 803us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:03.273472 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:03.273889 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 02:10:03.274344 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":444,"threads_started":1}
I20260926 02:10:03.279910 31455 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d2f8f2cdf8421098dac41087354e7f" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45111 } }
I20260926 02:10:03.280431 31455 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:10:03.280550 31455 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d2f8f2cdf8421098dac41087354e7f, State: Initialized, Role: FOLLOWER
I20260926 02:10:03.281227 31455 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d2f8f2cdf8421098dac41087354e7f" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45111 } }
I20260926 02:10:03.281468 31455 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:10:03.281599 31455 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:10:03.281786 31455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:10:03.284058 31455 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d2f8f2cdf8421098dac41087354e7f" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45111 } }
I20260926 02:10:03.284698 31455 leader_election.cc:302] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5d2f8f2cdf8421098dac41087354e7f; no voters:
I20260926 02:10:03.285384 31455 leader_election.cc:288] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:10:03.285571 31460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:10:03.286269 31460 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [term 1 LEADER]: Becoming Leader. State: Replica: c5d2f8f2cdf8421098dac41087354e7f, State: Running, Role: LEADER
I20260926 02:10:03.286827 31460 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d2f8f2cdf8421098dac41087354e7f" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45111 } }
I20260926 02:10:03.287637 31455 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:10:03.289968 31461 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5d2f8f2cdf8421098dac41087354e7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d2f8f2cdf8421098dac41087354e7f" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45111 } } }
I20260926 02:10:03.289861 31462 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5d2f8f2cdf8421098dac41087354e7f. Latest consensus state: current_term: 1 leader_uuid: "c5d2f8f2cdf8421098dac41087354e7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d2f8f2cdf8421098dac41087354e7f" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45111 } } }
I20260926 02:10:03.290369 31461 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [sys.catalog]: This master's current role is: LEADER
I20260926 02:10:03.290426 31462 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [sys.catalog]: This master's current role is: LEADER
I20260926 02:10:03.291152 31465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:10:03.296873 31465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:10:03.305456 31455 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 02:10:03.306252 31465 catalog_manager.cc:1347] Generated new cluster ID: 515d499c09174d4a99948cb49da27819
I20260926 02:10:03.306358 31465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:10:03.331104 31465 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:10:03.333282 31465 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:10:03.341886 31465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f: Generated new TSK 0
I20260926 02:10:03.343101 31465 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.159.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
I20260926 02:10:04.115563 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.159.148:37729/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:45111
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:10:04 dist-test-slave-03rn krb5kdc[30944](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790388604, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 02:10:04.585184 31500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:04.585611 31500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:04.585831 31500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:04.597736 31500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:04.597903 31500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:04.598148 31500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:10:04.613093 31500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.159.148:37729/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:45111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:04.614917 31500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:04.616858 31500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:04.637693 31510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:04.637755 31511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:04.645994 31513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:04.646333 31500 server_base.cc:1034] running on GCE node
I20260926 02:10:04.647146 31500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:04.648801 31500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:04.650112 31500 hybrid_clock.cc:648] HybridClock initialized: now 1790388604650022 us; error 86 us; skew 500 ppm
I20260926 02:10:04.650492 31500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:04 dist-test-slave-03rn krb5kdc[30944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388604, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:04.654297 31500 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:10:04.661268 31500 webserver.cc:460] Webserver started at http://127.18.159.129:35291/ using document root <none> and password file <none>
I20260926 02:10:04.662358 31500 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:04.662531 31500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:04.662995 31500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:10:04 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388604, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
I20260926 02:10:04.693994 31500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "5a93a1b11ee145f9bb578208730121f9"
format_stamp: "Formatted at 2026-09-26 02:10:04 on dist-test-slave-03rn"
server_key: "0d5b3f602a9f59513a8148e834ac68c2"
server_key_iv: "43d6171fec6ff2a1da8e3c2e822f00c9"
server_key_version: "kuduclusterkey@0"
I20260926 02:10:04.695478 31500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "5a93a1b11ee145f9bb578208730121f9"
format_stamp: "Formatted at 2026-09-26 02:10:04 on dist-test-slave-03rn"
server_key: "0d5b3f602a9f59513a8148e834ac68c2"
server_key_iv: "43d6171fec6ff2a1da8e3c2e822f00c9"
server_key_version: "kuduclusterkey@0"
I20260926 02:10:04.703722 31500 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.003s
I20260926 02:10:04.729784 31520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:04.731583 31500 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:10:04.731841 31500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "5a93a1b11ee145f9bb578208730121f9"
format_stamp: "Formatted at 2026-09-26 02:10:04 on dist-test-slave-03rn"
server_key: "0d5b3f602a9f59513a8148e834ac68c2"
server_key_iv: "43d6171fec6ff2a1da8e3c2e822f00c9"
server_key_version: "kuduclusterkey@0"
I20260926 02:10:04.732100 31500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:04.755275 31500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:04.761495 31500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:04.761997 31500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:04.766475 31500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:10:04.769043 31500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:10:04.769170 31500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:04.769377 31500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:10:04.769459 31500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:04.861927 31500 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:42373
I20260926 02:10:04.861925 31632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:42373 every 8 connection(s)
I20260926 02:10:04.864233 31500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:10:04.868709 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 31500
I20260926 02:10:04.868956 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:10:04.869930 19070 external_mini_cluster.cc:1428] Setting key 2771154a00b5737b10ab62c21e8642e8
Sep 26 02:10:04 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388604, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.159.130@KRBTEST.COM; defaulting to no policy
I20260926 02:10:04.906080 31635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:04.868028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:60695 (local address 127.18.159.190:45111)
0926 02:10:04.884810 (+ 16782us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:04.884817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:04.884845 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:10:04.885824 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:04.885831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:04.885941 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:04.886026 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:04.888531 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:04.889714 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:04.892263 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:04.892942 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:04.893046 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:04.897981 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:04.897989 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:04.897993 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:04.898039 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:04.901875 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:04.902995 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:04.903002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:04.903006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:04.903176 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:04.903747 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:04.903753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:04.903756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:04.904195 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:04.904335 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:04.905419 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0926 02:10:04.905667 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16625,"thread_start_us":16323,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.130@KRBTEST.COM" created.
I20260926 02:10:04.908783 31636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:04.868717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:45111 (local address 127.18.159.129:60695)
0926 02:10:04.881802 (+ 13085us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:04.881881 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:04.884821 (+ 2940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:04.886693 (+ 1872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:04.886709 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:04.887374 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:04.888280 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:04.888316 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:04.889947 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:04.889952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:04.892045 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:04.892063 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:04.893420 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:04.893437 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:04.893759 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:04.894625 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:04.894656 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:04.897644 (+ 2988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:04.902117 (+ 4473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:04.902128 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:04.902151 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:04.902816 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:04.903330 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:04.903335 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:04.903339 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:04.903598 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:04.904437 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:04.904446 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:04.904952 (+ 506us) client_negotiation.cc:769] Sending connection context
0926 02:10:04.906248 (+ 1296us) client_negotiation.cc:240] Negotiation successful
0926 02:10:04.907127 (+ 879us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12712,"thread_start_us":12449,"threads_started":1}
I20260926 02:10:04.911042 31633 heartbeater.cc:347] Connected to a master server at 127.18.159.190:45111
I20260926 02:10:04.911407 31633 heartbeater.cc:464] Registering TS with master...
I20260926 02:10:04.912380 31633 heartbeater.cc:511] Master 127.18.159.190:45111 requested a full tablet report, sending...
I20260926 02:10:04.915474 31420 ts_manager.cc:194] Registered new tserver with Master: 5a93a1b11ee145f9bb578208730121f9 (127.18.159.129:42373)
I20260926 02:10:04.917692 31420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:60695
Sep 26 02:10:04 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388604, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
I20260926 02:10:04.952564 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.159.148:37729/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:45111
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:10:05 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388604, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
W20260926 02:10:05.470858 31639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:05.471390 31639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:05.471594 31639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:05.484830 31639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:05.485051 31639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:05.485268 31639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:10:05.502128 31639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.159.148:37729/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:45111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:05.504132 31639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:05.506058 31639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:05.543964 31647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:05.552534 31650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:05.556555 31646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:05.556640 31639 server_base.cc:1034] running on GCE node
I20260926 02:10:05.557637 31639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:05.567451 31639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:05.569005 31639 hybrid_clock.cc:648] HybridClock initialized: now 1790388605568926 us; error 85 us; skew 500 ppm
I20260926 02:10:05.569571 31639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:05 dist-test-slave-03rn krb5kdc[30944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388605, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:05.579162 31639 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:10:05.616214 31639 webserver.cc:460] Webserver started at http://127.18.159.130:42429/ using document root <none> and password file <none>
I20260926 02:10:05.617410 31639 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:05.617666 31639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:05.618077 31639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:10:05 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388605, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
I20260926 02:10:05.650178 31639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "9e9acb1c608e4f02aceda2e7fabf0b2b"
format_stamp: "Formatted at 2026-09-26 02:10:05 on dist-test-slave-03rn"
server_key: "6916fab60d3be26f6f691e0352c3da3e"
server_key_iv: "fc86903ef06e58cffc1add4d20c53edd"
server_key_version: "kuduclusterkey@0"
I20260926 02:10:05.651667 31639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "9e9acb1c608e4f02aceda2e7fabf0b2b"
format_stamp: "Formatted at 2026-09-26 02:10:05 on dist-test-slave-03rn"
server_key: "6916fab60d3be26f6f691e0352c3da3e"
server_key_iv: "fc86903ef06e58cffc1add4d20c53edd"
server_key_version: "kuduclusterkey@0"
I20260926 02:10:05.669250 31639 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.006s sys 0.003s
I20260926 02:10:05.700598 31663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:05.702960 31639 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260926 02:10:05.703186 31639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "9e9acb1c608e4f02aceda2e7fabf0b2b"
format_stamp: "Formatted at 2026-09-26 02:10:05 on dist-test-slave-03rn"
server_key: "6916fab60d3be26f6f691e0352c3da3e"
server_key_iv: "fc86903ef06e58cffc1add4d20c53edd"
server_key_version: "kuduclusterkey@0"
I20260926 02:10:05.703608 31639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:05.726809 31639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:05.732375 31639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:05.732707 31639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:05.733929 31639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:10:05.735685 31639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:10:05.735775 31639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:05.735956 31639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:10:05.736027 31639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:05.839347 31639 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:45105
I20260926 02:10:05.839388 31775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:45105 every 8 connection(s)
I20260926 02:10:05.841351 31639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:10:05.847685 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 31639
I20260926 02:10:05.847859 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:10:05.848429 19070 external_mini_cluster.cc:1428] Setting key 433cd09c2711c8454543342978e9f014
Sep 26 02:10:05 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388605, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.159.131@KRBTEST.COM; defaulting to no policy
I20260926 02:10:05.885862 31779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:05.845428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:42871 (local address 127.18.159.190:45111)
0926 02:10:05.847388 (+ 1960us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:05.847395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:05.857665 (+ 10270us) server_negotiation.cc:407] Connection header received
0926 02:10:05.866035 (+ 8370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:05.866042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:05.866152 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:05.866289 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:05.868653 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:05.869755 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:05.872358 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:05.873020 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:05.873126 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:05.877993 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:05.877998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:05.878002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:05.878039 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:05.881652 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:05.882782 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:05.882789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:05.882792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:05.882961 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:05.883575 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:05.883581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:05.883584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:05.883970 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:05.884138 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:05.885195 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0926 02:10:05.885469 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1828,"thread_start_us":1726,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.131@KRBTEST.COM" created.
I20260926 02:10:05.887126 31778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:05.844892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:45111 (local address 127.18.159.130:42871)
0926 02:10:05.856952 (+ 12060us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:05.857030 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:05.865679 (+ 8649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:05.866617 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:05.866633 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:05.867359 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:05.868404 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:05.868440 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:05.870036 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:05.870046 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:05.872099 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:05.872117 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:05.873497 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:05.873504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:05.873790 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:05.874653 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:05.874685 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:05.877660 (+ 2975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:05.881905 (+ 4245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:05.881917 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:05.881939 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:05.882616 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:05.883114 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:05.883119 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:05.883123 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:05.883417 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:05.884126 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:05.884135 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:05.884733 (+ 598us) client_negotiation.cc:769] Sending connection context
0926 02:10:05.885063 (+ 330us) client_negotiation.cc:240] Negotiation successful
0926 02:10:05.885637 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11687,"thread_start_us":11435,"threads_started":1}
I20260926 02:10:05.889400 31776 heartbeater.cc:347] Connected to a master server at 127.18.159.190:45111
I20260926 02:10:05.889746 31776 heartbeater.cc:464] Registering TS with master...
I20260926 02:10:05.890686 31776 heartbeater.cc:511] Master 127.18.159.190:45111 requested a full tablet report, sending...
I20260926 02:10:05.892587 31420 ts_manager.cc:194] Registered new tserver with Master: 9e9acb1c608e4f02aceda2e7fabf0b2b (127.18.159.130:45105)
I20260926 02:10:05.893658 31420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:42871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
I20260926 02:10:05.921478 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.159.148:37729/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:45111
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:10:05.925230 31633 heartbeater.cc:503] Master 127.18.159.190:45111 was elected leader, sending a full tablet report...
W20260926 02:10:06.277757 31782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:06.278199 31782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:06.278373 31782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:06.287717 31782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:06.287854 31782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:06.287971 31782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:10:06.298641 31782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.18.159.148:37729/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:45111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:06.300019 31782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:06.301462 31782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:06.315616 31790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:06.315701 31787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:06.315701 31788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:06.316897 31782 server_base.cc:1034] running on GCE node
I20260926 02:10:06.317701 31782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:06.318727 31782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:06.319973 31782 hybrid_clock.cc:648] HybridClock initialized: now 1790388606319927 us; error 48 us; skew 500 ppm
I20260926 02:10:06.320399 31782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:06 dist-test-slave-03rn krb5kdc[30944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388606, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:06.323643 31782 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:10:06.325409 31782 webserver.cc:460] Webserver started at http://127.18.159.131:42277/ using document root <none> and password file <none>
I20260926 02:10:06.326157 31782 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:06.326311 31782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:06.326740 31782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:10:06 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388606, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
I20260926 02:10:06.343174 31782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "a375b632ef6d4309be928de09a0288ff"
format_stamp: "Formatted at 2026-09-26 02:10:06 on dist-test-slave-03rn"
server_key: "d525f1f33950c4fe743fe35367fddb77"
server_key_iv: "ea6dbd37d7cc6b0af922a467291d7d93"
server_key_version: "kuduclusterkey@0"
I20260926 02:10:06.344427 31782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "a375b632ef6d4309be928de09a0288ff"
format_stamp: "Formatted at 2026-09-26 02:10:06 on dist-test-slave-03rn"
server_key: "d525f1f33950c4fe743fe35367fddb77"
server_key_iv: "ea6dbd37d7cc6b0af922a467291d7d93"
server_key_version: "kuduclusterkey@0"
I20260926 02:10:06.350508 31782 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:10:06.365813 31797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:06.368148 31782 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:10:06.368423 31782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "a375b632ef6d4309be928de09a0288ff"
format_stamp: "Formatted at 2026-09-26 02:10:06 on dist-test-slave-03rn"
server_key: "d525f1f33950c4fe743fe35367fddb77"
server_key_iv: "ea6dbd37d7cc6b0af922a467291d7d93"
server_key_version: "kuduclusterkey@0"
I20260926 02:10:06.368722 31782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:06.392984 31782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:06.397724 31782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:06.398058 31782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:06.399091 31782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:10:06.400615 31782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:10:06.400695 31782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:06.400830 31782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:10:06.400880 31782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:06.436434 31782 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:35361
I20260926 02:10:06.436504 31909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:35361 every 8 connection(s)
I20260926 02:10:06.437885 31782 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:10:06.439028 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 31782
I20260926 02:10:06.439193 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:10:06.439788 19070 external_mini_cluster.cc:1428] Setting key ff0fdbd9137aeed45e15c9794dd7f15d
Sep 26 02:10:06 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388606, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:10:06.461833 31912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:06.441177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:33325 (local address 127.18.159.190:45111)
0926 02:10:06.445854 (+ 4677us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:06.445860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:06.445884 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:10:06.447089 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:06.447094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:06.447191 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:06.447324 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:06.448944 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:06.449734 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:06.451419 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:06.451914 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:06.452074 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:06.456145 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:06.456152 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:06.456179 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:06.456221 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:06.458787 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:06.459570 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:06.459574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:06.459577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:06.459708 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:06.460135 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:06.460138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:06.460140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:06.460496 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:06.460611 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:06.461378 (+ 767us) server_negotiation.cc:299] Negotiation successful
0926 02:10:06.461565 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4528,"thread_start_us":4362,"threads_started":1}
I20260926 02:10:06.462898 31913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:06.441968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:45111 (local address 127.18.159.131:33325)
0926 02:10:06.444749 (+ 2781us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:06.444798 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:06.446769 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:06.447598 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:06.447607 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:06.448057 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:06.448763 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:06.448783 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:06.449918 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:06.449922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:06.451253 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:06.451266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:06.452208 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:06.452216 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:06.452476 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:06.453182 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:06.453206 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:06.455874 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:06.458950 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:06.458958 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:06.458973 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:06.459446 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:06.459802 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:06.459805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:06.459807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:06.460032 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:06.460646 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:06.460652 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:06.461029 (+ 377us) client_negotiation.cc:769] Sending connection context
0926 02:10:06.461286 (+ 257us) client_negotiation.cc:240] Negotiation successful
0926 02:10:06.461696 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2512,"thread_start_us":2239,"threads_started":1}
I20260926 02:10:06.464493 31910 heartbeater.cc:347] Connected to a master server at 127.18.159.190:45111
I20260926 02:10:06.464799 31910 heartbeater.cc:464] Registering TS with master...
I20260926 02:10:06.465442 31910 heartbeater.cc:511] Master 127.18.159.190:45111 requested a full tablet report, sending...
I20260926 02:10:06.467099 31420 ts_manager.cc:194] Registered new tserver with Master: a375b632ef6d4309be928de09a0288ff (127.18.159.131:35361)
I20260926 02:10:06.468109 31420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:33325
I20260926 02:10:06.468885 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:10:06.470564 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 31500
Sep 26 02:10:06 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:10:06.488824 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:06.471513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59130 (local address 127.18.159.129:42373)
0926 02:10:06.472144 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:06.472185 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:06.472220 (+ 35us) server_negotiation.cc:407] Connection header received
0926 02:10:06.472405 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:06.472412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:06.472633 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:06.472819 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:06.474517 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:06.475676 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:06.477321 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:06.477924 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:06.478078 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:06.482695 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:06.482709 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:06.482725 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:06.482765 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:06.485704 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:06.486520 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:06.486528 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:06.486535 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:06.486682 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:06.487061 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:06.487064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:06.487066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:06.487486 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:06.487756 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:06.488271 (+ 515us) server_negotiation.cc:299] Negotiation successful
0926 02:10:06.488539 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":271,"threads_started":1}
I20260926 02:10:06.490176 31628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:06.575599 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 31500
I20260926 02:10:06.591899 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 31639
Sep 26 02:10:06 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.130@KRBTEST.COM
I20260926 02:10:06.609946 31917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:06.592837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51542 (local address 127.18.159.130:45105)
0926 02:10:06.593380 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:06.593388 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:06.593471 (+ 83us) server_negotiation.cc:407] Connection header received
0926 02:10:06.593541 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:06.593548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:06.593783 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:06.593960 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:06.595411 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:06.596503 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:06.597941 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:06.598529 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:06.598679 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:06.603392 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:06.603405 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:06.603421 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:06.603475 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:06.606381 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:06.607345 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:06.607353 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:06.607355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:06.607516 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:06.607998 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:06.608002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:06.608004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:06.608581 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:06.608894 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:06.609248 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 02:10:06.609536 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":298,"threads_started":1}
I20260926 02:10:06.611279 31771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:06.695940 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 31639
I20260926 02:10:06.716038 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 31782
Sep 26 02:10:06 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.131@KRBTEST.COM
I20260926 02:10:06.733498 31919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:06.717245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.18.159.131:35361)
0926 02:10:06.717824 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:06.717831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:06.717861 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:10:06.717918 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:06.717924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:06.718144 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:06.718276 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:06.719579 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:06.720770 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:06.722153 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:06.722739 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:06.722840 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:06.727338 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:06.727353 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:06.727369 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:06.727412 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:06.730613 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:06.731385 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:06.731392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:06.731394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:06.731541 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:06.731954 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:06.731958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:06.731960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:06.732426 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:06.732683 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:06.732952 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 02:10:06.733200 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":290,"threads_started":1}
I20260926 02:10:06.734876 31905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:06.818718 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 31782
I20260926 02:10:06.834398 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 31388
I20260926 02:10:06.835750 31450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:06.943830 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 31388
I20260926 02:10:06.967996 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:45111
--webserver_interface=127.18.159.190
--webserver_port=35621
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:45111
--ranger_config_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.159.148:37729/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.159.148:37729/kms with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:10:07.313810 31922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:07.314136 31922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:07.314306 31922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:07.322000 31922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:10:07.322129 31922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:07.322220 31922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:07.322310 31922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:10:07.322381 31922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:10:07.333645 31922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.159.148:37729/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:45111
--ranger_config_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:45111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=35621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:07.335059 31922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:07.336510 31922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:07.349255 31927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:07.349340 31928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:07.349128 31930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:07.353915 31922 server_base.cc:1034] running on GCE node
I20260926 02:10:07.354864 31922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:07.356614 31922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:07.357972 31922 hybrid_clock.cc:648] HybridClock initialized: now 1790388607357900 us; error 62 us; skew 500 ppm
I20260926 02:10:07.358458 31922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:07 dist-test-slave-03rn krb5kdc[30944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388607, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:07.361922 31922 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:10:07.363585 31922 webserver.cc:460] Webserver started at http://127.18.159.190:35621/ using document root <none> and password file <none>
I20260926 02:10:07.364382 31922 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:07.364545 31922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 02:10:07 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388607, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
I20260926 02:10:07.438532 31922 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:10:07.442163 31938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:07.443908 31922 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 02:10:07.444209 31922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "c5d2f8f2cdf8421098dac41087354e7f"
format_stamp: "Formatted at 2026-09-26 02:10:03 on dist-test-slave-03rn"
server_key: "807510ca7d4febb3c8a3d551dd591288"
server_key_iv: "2187cfadbadcc5b253147bdcef6ee688"
server_key_version: "kuduclusterkey@0"
I20260926 02:10:07.445322 31922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:07.463622 31922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:07.468292 31922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:07.468597 31922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:07.489420 31922 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:45111
I20260926 02:10:07.489441 31989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:45111 every 8 connection(s)
I20260926 02:10:07.491181 31922 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:10:07.493312 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 31922
I20260926 02:10:07.494248 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.159.148:37729/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:42373
--local_ip_for_outbound_sockets=127.18.159.129
--tserver_master_addrs=127.18.159.190:45111
--webserver_port=35291
--webserver_interface=127.18.159.129
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.159.148:37729/kms with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:10:07.498855 31990 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:10:07.502475 31990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f: Bootstrap starting.
I20260926 02:10:07.522347 31990 log.cc:824] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f: Log is configured to *not* fsync() on all Append() calls
I20260926 02:10:07.531812 31990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:10:07.532495 31990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f: Bootstrap complete.
I20260926 02:10:07.541993 31990 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d2f8f2cdf8421098dac41087354e7f" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45111 } }
I20260926 02:10:07.543043 31990 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d2f8f2cdf8421098dac41087354e7f, State: Initialized, Role: FOLLOWER
I20260926 02:10:07.543733 31990 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d2f8f2cdf8421098dac41087354e7f" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45111 } }
I20260926 02:10:07.543992 31990 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:10:07.544133 31990 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:10:07.544366 31990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:10:07.548621 31990 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d2f8f2cdf8421098dac41087354e7f" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45111 } }
I20260926 02:10:07.549293 31990 leader_election.cc:302] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5d2f8f2cdf8421098dac41087354e7f; no voters:
I20260926 02:10:07.550014 31990 leader_election.cc:288] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 02:10:07.550163 31994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [term 2 FOLLOWER]: Leader election won for term 2
I20260926 02:10:07.552661 31990 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:10:07.557385 31994 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [term 2 LEADER]: Becoming Leader. State: Replica: c5d2f8f2cdf8421098dac41087354e7f, State: Running, Role: LEADER
I20260926 02:10:07.558218 31994 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d2f8f2cdf8421098dac41087354e7f" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45111 } }
I20260926 02:10:07.562517 31990 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 02:10:07.562479 31995 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c5d2f8f2cdf8421098dac41087354e7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d2f8f2cdf8421098dac41087354e7f" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45111 } } }
I20260926 02:10:07.562876 31995 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [sys.catalog]: This master's current role is: LEADER
I20260926 02:10:07.575316 31996 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5d2f8f2cdf8421098dac41087354e7f. Latest consensus state: current_term: 2 leader_uuid: "c5d2f8f2cdf8421098dac41087354e7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d2f8f2cdf8421098dac41087354e7f" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45111 } } }
I20260926 02:10:07.575773 31996 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f [sys.catalog]: This master's current role is: LEADER
I20260926 02:10:07.576690 32009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:10:07.588950 32009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:10:07.590924 32009 catalog_manager.cc:1259] Loaded cluster ID: 515d499c09174d4a99948cb49da27819
I20260926 02:10:07.591197 32009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:10:07.597494 32009 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:10:07.600359 32009 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f: Loaded TSK: 0
I20260926 02:10:07.602818 32009 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 02:10:07.987362 31992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:07.987905 31992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:07.988097 31992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:08.000553 31992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:08.000725 31992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:08.000979 31992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:10:08.017926 31992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.159.148:37729/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:42373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=35291
--tserver_master_addrs=127.18.159.190:45111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:08.020028 31992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:08.022049 31992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:08.057487 32028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:08.057849 32027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:08.067040 31992 server_base.cc:1034] running on GCE node
W20260926 02:10:08.067574 32030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:08.068397 31992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:08.069700 31992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:08.070976 31992 hybrid_clock.cc:648] HybridClock initialized: now 1790388608070896 us; error 78 us; skew 500 ppm
I20260926 02:10:08.071342 31992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:08 dist-test-slave-03rn krb5kdc[30944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388608, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:08.075028 31992 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:10:08.081676 31992 webserver.cc:460] Webserver started at http://127.18.159.129:35291/ using document root <none> and password file <none>
I20260926 02:10:08.082494 31992 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:08.082687 31992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 02:10:08 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388608, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
I20260926 02:10:08.146459 31992 fs_manager.cc:709] Time spent opening directory manager: real 0.026s user 0.013s sys 0.012s
I20260926 02:10:08.156919 32039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:08.159541 31992 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.008s sys 0.001s
I20260926 02:10:08.159906 31992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "5a93a1b11ee145f9bb578208730121f9"
format_stamp: "Formatted at 2026-09-26 02:10:04 on dist-test-slave-03rn"
server_key: "0d5b3f602a9f59513a8148e834ac68c2"
server_key_iv: "43d6171fec6ff2a1da8e3c2e822f00c9"
server_key_version: "kuduclusterkey@0"
I20260926 02:10:08.161664 31992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:08.200037 31992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:08.205089 31992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:08.205438 31992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:08.206789 31992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:10:08.208765 31992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:10:08.208973 31992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 02:10:08.209167 31992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:10:08.209317 31992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:08.281853 31992 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:42373
I20260926 02:10:08.281900 32151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:42373 every 8 connection(s)
I20260926 02:10:08.283452 31992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:10:08.286177 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 31992
I20260926 02:10:08.287029 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.159.148:37729/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:45105
--local_ip_for_outbound_sockets=127.18.159.130
--tserver_master_addrs=127.18.159.190:45111
--webserver_port=42429
--webserver_interface=127.18.159.130
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.159.148:37729/kms with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:10:08 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388608, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:10:08.348935 32155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:08.287094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:45111 (local address 127.18.159.129:33695)
0926 02:10:08.319858 (+ 32764us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:08.319952 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:08.322987 (+ 3035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:08.326650 (+ 3663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:08.326662 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:08.327320 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:08.328124 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:08.328150 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:08.330800 (+ 2650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:08.330806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:08.332923 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:08.332943 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:08.334859 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:08.334866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:08.335176 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:08.336076 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:08.336117 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:08.338849 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:08.343847 (+ 4998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:08.343857 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:08.343872 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:08.344531 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:08.345104 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:08.345109 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:08.345113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:08.345381 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:08.346399 (+ 1018us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:08.346406 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:08.346778 (+ 372us) client_negotiation.cc:769] Sending connection context
0926 02:10:08.347078 (+ 300us) client_negotiation.cc:240] Negotiation successful
0926 02:10:08.347607 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32330,"thread_start_us":36206,"threads_started":1}
I20260926 02:10:08.349080 32154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:08.287081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:33695 (local address 127.18.159.190:45111)
0926 02:10:08.304796 (+ 17715us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:08.304822 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:08.320187 (+ 15365us) server_negotiation.cc:407] Connection header received
0926 02:10:08.324809 (+ 4622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:08.324833 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:08.325389 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:08.325794 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:08.328423 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:08.330533 (+ 2110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:08.333126 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:08.334240 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:08.334411 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:08.339258 (+ 4847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:08.339271 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:08.339293 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:08.339349 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:08.343616 (+ 4267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:08.344728 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:08.344740 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:08.344750 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:08.344970 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:08.345534 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:08.345540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:08.345544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:08.346245 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:08.346474 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:08.347349 (+ 875us) server_negotiation.cc:299] Negotiation successful
0926 02:10:08.347794 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17418,"thread_start_us":17192,"threads_started":1}
I20260926 02:10:08.353715 32152 heartbeater.cc:347] Connected to a master server at 127.18.159.190:45111
I20260926 02:10:08.354074 32152 heartbeater.cc:464] Registering TS with master...
I20260926 02:10:08.355062 32152 heartbeater.cc:511] Master 127.18.159.190:45111 requested a full tablet report, sending...
W20260926 02:10:08.358816 32152 heartbeater.cc:644] Failed to heartbeat to 127.18.159.190:45111 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 02:10:08.359067 32152 heartbeater.cc:464] Registering TS with master...
I20260926 02:10:08.359421 32152 heartbeater.cc:511] Master 127.18.159.190:45111 requested a full tablet report, sending...
I20260926 02:10:08.360954 32152 heartbeater.cc:464] Registering TS with master...
I20260926 02:10:08.361534 32152 heartbeater.cc:511] Master 127.18.159.190:45111 requested a full tablet report, sending...
W20260926 02:10:08.363067 32152 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:10:08.711194 32156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:08.711573 32156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:08.711750 32156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:08.719748 32156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:08.719934 32156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:08.720116 32156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:10:08.731580 32156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.159.148:37729/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:45105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=42429
--tserver_master_addrs=127.18.159.190:45111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:08.733095 32156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:08.734495 32156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:08.763662 32171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:08.772011 32173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:08.773034 32170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:08.775303 32156 server_base.cc:1034] running on GCE node
I20260926 02:10:08.776273 32156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:08.781827 32156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:08.783596 32156 hybrid_clock.cc:648] HybridClock initialized: now 1790388608783521 us; error 69 us; skew 500 ppm
I20260926 02:10:08.784204 32156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:08 dist-test-slave-03rn krb5kdc[30944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388608, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:08.788923 32156 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:10:08.808647 32156 webserver.cc:460] Webserver started at http://127.18.159.130:42429/ using document root <none> and password file <none>
I20260926 02:10:08.809934 32156 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:08.810215 32156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 02:10:08 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388608, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
I20260926 02:10:08.844597 32156 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.007s sys 0.005s
I20260926 02:10:08.849040 32183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:08.851292 32156 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:10:08.851528 32156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "9e9acb1c608e4f02aceda2e7fabf0b2b"
format_stamp: "Formatted at 2026-09-26 02:10:05 on dist-test-slave-03rn"
server_key: "6916fab60d3be26f6f691e0352c3da3e"
server_key_iv: "fc86903ef06e58cffc1add4d20c53edd"
server_key_version: "kuduclusterkey@0"
I20260926 02:10:08.853106 32156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 26 02:10:08 dist-test-slave-03rn krb5kdc[30944](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790388608, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:08.899320 32156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:08.907896 32156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:08.908497 32156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:08.913928 32156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:10:08.915977 32156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:10:08.916142 32156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 02:10:08.916297 32156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:10:08.916393 32156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:08.977347 32156 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:45105
I20260926 02:10:08.977412 32296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:45105 every 8 connection(s)
I20260926 02:10:08.979427 32156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:10:08.979985 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 32156
I20260926 02:10:08.981019 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.159.148:37729/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:35361
--local_ip_for_outbound_sockets=127.18.159.131
--tserver_master_addrs=127.18.159.190:45111
--webserver_port=42277
--webserver_interface=127.18.159.131
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.159.148:37729/kms with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:10:09 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388608, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:10:09.024051 32299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:08.989757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:45111 (local address 127.18.159.130:39531)
0926 02:10:08.990921 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:08.990977 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:08.997336 (+ 6359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:08.999503 (+ 2167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:08.999520 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:09.000298 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:09.001430 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:09.001468 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:09.003239 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:09.003248 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:09.005705 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:09.005731 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:09.007355 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:09.007365 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:09.008589 (+ 1224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:09.009604 (+ 1015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:09.009649 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:09.013886 (+ 4237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:09.018801 (+ 4915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:09.018815 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:09.018837 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:09.019554 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:09.020041 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:09.020046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:09.020050 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:09.020397 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:09.021111 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:09.021121 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:09.021602 (+ 481us) client_negotiation.cc:769] Sending connection context
0926 02:10:09.021903 (+ 301us) client_negotiation.cc:240] Negotiation successful
0926 02:10:09.022520 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"thread_start_us":655,"threads_started":1}
I20260926 02:10:09.024789 32301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:08.990358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:39531 (local address 127.18.159.190:45111)
0926 02:10:08.998690 (+ 8332us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:08.998696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:08.998735 (+ 39us) server_negotiation.cc:407] Connection header received
0926 02:10:08.998806 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:08.998812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:08.998904 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:08.999169 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:09.001732 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:09.002998 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:09.005979 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:09.006775 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:09.006935 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:09.014303 (+ 7368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:09.014314 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:09.014320 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:09.014378 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:09.018551 (+ 4173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:09.019735 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:09.019741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:09.019745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:09.019906 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:09.020536 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:09.020541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:09.020545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:09.020901 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:09.021041 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:09.024050 (+ 3009us) server_negotiation.cc:299] Negotiation successful
0926 02:10:09.024386 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8137,"thread_start_us":7959,"threads_started":1}
I20260926 02:10:09.026430 32297 heartbeater.cc:347] Connected to a master server at 127.18.159.190:45111
I20260926 02:10:09.026835 32297 heartbeater.cc:464] Registering TS with master...
I20260926 02:10:09.027938 32297 heartbeater.cc:511] Master 127.18.159.190:45111 requested a full tablet report, sending...
I20260926 02:10:09.030352 31955 ts_manager.cc:194] Registered new tserver with Master: 9e9acb1c608e4f02aceda2e7fabf0b2b (127.18.159.130:45105)
I20260926 02:10:09.032697 31955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:39531
I20260926 02:10:09.363255 32152 heartbeater.cc:464] Registering TS with master...
I20260926 02:10:09.363722 32152 heartbeater.cc:511] Master 127.18.159.190:45111 requested a full tablet report, sending...
I20260926 02:10:09.365003 31955 ts_manager.cc:194] Registered new tserver with Master: 5a93a1b11ee145f9bb578208730121f9 (127.18.159.129:42373)
I20260926 02:10:09.366263 31955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:33695
Sep 26 02:10:09 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388608, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
W20260926 02:10:09.467211 32300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:09.467749 32300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:09.467952 32300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:09.480589 32300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:09.480836 32300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:09.481061 32300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:10:09.508513 32300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.159.148:37729/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:35361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=42277
--tserver_master_addrs=127.18.159.190:45111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:09.510628 32300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:09.512555 32300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:09.552321 32306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:09.555297 32307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:09.555498 32300 server_base.cc:1034] running on GCE node
W20260926 02:10:09.564560 32309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:09.565640 32300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:09.579041 32300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:09.580534 32300 hybrid_clock.cc:648] HybridClock initialized: now 1790388609580298 us; error 128 us; skew 500 ppm
I20260926 02:10:09.581038 32300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:09 dist-test-slave-03rn krb5kdc[30944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388609, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:09.585376 32300 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:10:09.614652 32300 webserver.cc:460] Webserver started at http://127.18.159.131:42277/ using document root <none> and password file <none>
I20260926 02:10:09.615804 32300 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:09.615993 32300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 02:10:09 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388609, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
I20260926 02:10:09.648710 32300 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.007s sys 0.000s
I20260926 02:10:09.655144 32317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:09.657498 32300 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.003s
I20260926 02:10:09.657737 32300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "a375b632ef6d4309be928de09a0288ff"
format_stamp: "Formatted at 2026-09-26 02:10:06 on dist-test-slave-03rn"
server_key: "d525f1f33950c4fe743fe35367fddb77"
server_key_iv: "ea6dbd37d7cc6b0af922a467291d7d93"
server_key_version: "kuduclusterkey@0"
I20260926 02:10:09.659274 32300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:09.680234 32300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:09.685842 32300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:09.686209 32300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:09.687552 32300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:10:09.689661 32300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:10:09.689816 32300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 02:10:09.689956 32300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:10:09.690080 32300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:10:09 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388608, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for HTTP/127.18.159.148@KRBTEST.COM
I20260926 02:10:09.909942 32300 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:35361
I20260926 02:10:09.910033 32430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:35361 every 8 connection(s)
I20260926 02:10:09.911991 32300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:10:09.916363 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 32300
I20260926 02:10:09.953619 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:09.917507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57342 (local address 127.18.159.190:45111)
0926 02:10:09.928727 (+ 11220us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:09.928732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:09.931557 (+ 2825us) server_negotiation.cc:407] Connection header received
0926 02:10:09.933066 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:09.933073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:09.933182 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:09.933282 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:09.936990 (+ 3708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:09.938180 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:09.940481 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:09.941218 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:09.941346 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:09.945422 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:09.945431 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:09.945435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:09.945499 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:09.949444 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:09.950893 (+ 1449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:09.950901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:09.950905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:09.951100 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:09.951619 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:09.951625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:09.951628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:09.952006 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:09.952298 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:09.952904 (+ 606us) server_negotiation.cc:299] Negotiation successful
0926 02:10:09.953222 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11095,"thread_start_us":10976,"threads_started":1}
Sep 26 02:10:09 dist-test-slave-03rn krb5kdc[30944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388609, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:10:09.979005 32436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:09.940849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:36911 (local address 127.18.159.190:45111)
0926 02:10:09.944709 (+ 3860us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:09.944720 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:09.954513 (+ 9793us) server_negotiation.cc:407] Connection header received
0926 02:10:09.958195 (+ 3682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:09.958201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:09.958327 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:09.958437 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:09.960886 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:09.962082 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:09.964714 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:09.965461 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:09.965613 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:09.970893 (+ 5280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:09.970900 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:09.970905 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:09.970953 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:09.974657 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:09.975728 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:09.975734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:09.975737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:09.975996 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:09.976751 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:09.976758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:09.976761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:09.977158 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:09.977310 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:09.978295 (+ 985us) server_negotiation.cc:299] Negotiation successful
0926 02:10:09.978617 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3674,"thread_start_us":3482,"threads_started":1}
I20260926 02:10:09.984562 32435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:09.938318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:45111 (local address 127.18.159.131:36911)
0926 02:10:09.954336 (+ 16018us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:09.954395 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:09.957818 (+ 3423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:09.958775 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:09.958789 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:09.959586 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:09.960629 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:09.960661 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:09.962307 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:09.962314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:09.964517 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:09.964541 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:09.965954 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:09.965962 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:09.966281 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:09.967133 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:09.967169 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:09.970627 (+ 3458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:09.974883 (+ 4256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:09.974895 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:09.974914 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:09.975567 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:09.976264 (+ 697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:09.976269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:09.976276 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:09.976597 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:09.977426 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:09.977435 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:09.977861 (+ 426us) client_negotiation.cc:769] Sending connection context
0926 02:10:09.979635 (+ 1774us) client_negotiation.cc:240] Negotiation successful
0926 02:10:09.980276 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15636,"thread_start_us":14548,"threads_started":1}
I20260926 02:10:09.986583 32431 heartbeater.cc:347] Connected to a master server at 127.18.159.190:45111
I20260926 02:10:09.986951 32431 heartbeater.cc:464] Registering TS with master...
I20260926 02:10:09.987908 32431 heartbeater.cc:511] Master 127.18.159.190:45111 requested a full tablet report, sending...
I20260926 02:10:09.989719 31955 ts_manager.cc:194] Registered new tserver with Master: a375b632ef6d4309be928de09a0288ff (127.18.159.131:35361)
I20260926 02:10:09.990818 31955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:36911
I20260926 02:10:09.996048 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:10:10.022039 32436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:09.999907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57358 (local address 127.18.159.190:45111)
0926 02:10:10.000118 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:10.000127 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:10.005530 (+ 5403us) server_negotiation.cc:407] Connection header received
0926 02:10:10.005739 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:10.005745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:10.005837 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:10.005958 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:10.008669 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.009775 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.011251 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.011847 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.012011 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:10.015155 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:10.015162 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:10.015167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:10.015219 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:10.018956 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:10.019934 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:10.019939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:10.019943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:10.020120 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:10.020534 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:10.020539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:10.020542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:10.020903 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:10.021151 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:10.021444 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 02:10:10.021690 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:10:10.028523 31955 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 02:10:10.038199 32297 heartbeater.cc:503] Master 127.18.159.190:45111 was elected leader, sending a full tablet report...
W20260926 02:10:10.040093 31955 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:10:10.099681 32455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:10.059799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41784 (local address 127.18.159.129:42373)
0926 02:10:10.086242 (+ 26443us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:10.086256 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:10.088322 (+ 2066us) server_negotiation.cc:407] Connection header received
0926 02:10:10.088475 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:10.088486 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:10.088899 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:10.089299 (+ 400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:10:10.090881 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.092692 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.096010 (+ 3318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.097814 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.098036 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:10.098228 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:10.099130 (+ 902us) server_negotiation.cc:299] Negotiation successful
0926 02:10:10.099323 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26181,"thread_start_us":26463,"threads_started":1}
I20260926 02:10:10.100634 32454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:10.059451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:42373 (local address 127.0.0.1:41784)
0926 02:10:10.087687 (+ 28236us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:10.087733 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:10.087970 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:10.089791 (+ 1821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:10.089805 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:10.089855 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:10:10.090537 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:10.090558 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.093171 (+ 2613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.093180 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:10.095773 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:10.095786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.098407 (+ 2621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.098414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:10.098650 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:10.098710 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 02:10:10.098857 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 02:10:10.100133 (+ 1276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27939,"mutex_wait_us":980,"thread_start_us":33051,"threads_started":1}
I20260926 02:10:10.105089 32457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:10.063729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:45105 (local address 127.0.0.1:38748)
0926 02:10:10.093554 (+ 29825us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:10.093573 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:10.093698 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:10.095781 (+ 2083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:10.095786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:10.095815 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:10:10.096236 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:10.096248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.098510 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.098516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:10.102158 (+ 3642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:10.102181 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.104225 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.104236 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:10.104411 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:10.104447 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:10:10.104545 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 02:10:10.104697 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29684,"mutex_wait_us":58,"thread_start_us":35654,"threads_started":1}
I20260926 02:10:10.105146 32458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:10.064472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38748 (local address 127.18.159.130:45105)
0926 02:10:10.092107 (+ 27635us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:10.092118 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:10.093836 (+ 1718us) server_negotiation.cc:407] Connection header received
0926 02:10:10.093963 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:10.093973 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:10.094279 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:10.094505 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:10:10.096417 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.098303 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.102365 (+ 4062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.103901 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.104131 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:10.104366 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:10.104638 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 02:10:10.104807 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27426,"thread_start_us":27663,"threads_started":1}
I20260926 02:10:10.105123 32087 tablet_service.cc:1467] Processing CreateTablet for tablet e1e15c1f9d4e40d9b3bd6cfb4f81fd6c (DEFAULT_TABLE table=test-table [id=1a6a7a9f32ce47f0b896414a17523aef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:10:10.106618 32087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e15c1f9d4e40d9b3bd6cfb4f81fd6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:10:10.111981 32453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:10.059606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:35361 (local address 127.0.0.1:53992)
0926 02:10:10.092840 (+ 33234us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:10.092871 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:10.093025 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:10.096417 (+ 3392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:10.096422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:10.096449 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:10:10.096837 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:10.096849 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.098850 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.098857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:10.108493 (+ 9636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:10.108516 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.110892 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.110899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:10.111244 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:10.111286 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 02:10:10.111396 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 02:10:10.111566 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33058,"mutex_wait_us":874,"thread_start_us":27641,"threads_started":1}
I20260926 02:10:10.113039 32456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:10.060577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.18.159.131:35361)
0926 02:10:10.087029 (+ 26452us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:10.087040 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:10.094774 (+ 7734us) server_negotiation.cc:407] Connection header received
0926 02:10:10.095006 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:10.095015 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:10.095257 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:10.095447 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:10:10.097076 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.098660 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.108730 (+ 10070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.110276 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.110482 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:10.112407 (+ 1925us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:10.112531 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 02:10:10.112707 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26237,"thread_start_us":32255,"threads_started":1}
I20260926 02:10:10.112977 32231 tablet_service.cc:1467] Processing CreateTablet for tablet e1e15c1f9d4e40d9b3bd6cfb4f81fd6c (DEFAULT_TABLE table=test-table [id=1a6a7a9f32ce47f0b896414a17523aef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:10:10.114759 32231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e15c1f9d4e40d9b3bd6cfb4f81fd6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:10:10.117940 32365 tablet_service.cc:1467] Processing CreateTablet for tablet e1e15c1f9d4e40d9b3bd6cfb4f81fd6c (DEFAULT_TABLE table=test-table [id=1a6a7a9f32ce47f0b896414a17523aef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:10:10.120195 32365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e15c1f9d4e40d9b3bd6cfb4f81fd6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:10:10.138388 32460 tablet_bootstrap.cc:492] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b: Bootstrap starting.
I20260926 02:10:10.140856 32459 tablet_bootstrap.cc:492] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9: Bootstrap starting.
I20260926 02:10:10.144513 32460 tablet_bootstrap.cc:654] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b: Neither blocks nor log segments found. Creating new log.
I20260926 02:10:10.146073 32460 log.cc:824] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b: Log is configured to *not* fsync() on all Append() calls
I20260926 02:10:10.146374 32459 tablet_bootstrap.cc:654] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9: Neither blocks nor log segments found. Creating new log.
I20260926 02:10:10.148191 32459 log.cc:824] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9: Log is configured to *not* fsync() on all Append() calls
I20260926 02:10:10.154130 32461 tablet_bootstrap.cc:492] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P a375b632ef6d4309be928de09a0288ff: Bootstrap starting.
I20260926 02:10:10.158133 32459 tablet_bootstrap.cc:492] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9: No bootstrap required, opened a new log
I20260926 02:10:10.158684 32459 ts_tablet_manager.cc:1397] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.000s
I20260926 02:10:10.159601 32461 tablet_bootstrap.cc:654] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P a375b632ef6d4309be928de09a0288ff: Neither blocks nor log segments found. Creating new log.
I20260926 02:10:10.161259 32461 log.cc:824] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P a375b632ef6d4309be928de09a0288ff: Log is configured to *not* fsync() on all Append() calls
I20260926 02:10:10.162267 32460 tablet_bootstrap.cc:492] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b: No bootstrap required, opened a new log
I20260926 02:10:10.162778 32460 ts_tablet_manager.cc:1397] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.005s
I20260926 02:10:10.173792 32460 raft_consensus.cc:348] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a93a1b11ee145f9bb578208730121f9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42373 } } peers { permanent_uuid: "a375b632ef6d4309be928de09a0288ff" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35361 } } peers { permanent_uuid: "9e9acb1c608e4f02aceda2e7fabf0b2b" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45105 } }
I20260926 02:10:10.174412 32460 raft_consensus.cc:374] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:10:10.174562 32460 raft_consensus.cc:729] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e9acb1c608e4f02aceda2e7fabf0b2b, State: Initialized, Role: FOLLOWER
I20260926 02:10:10.175607 32460 consensus_queue.cc:260] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a93a1b11ee145f9bb578208730121f9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42373 } } peers { permanent_uuid: "a375b632ef6d4309be928de09a0288ff" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35361 } } peers { permanent_uuid: "9e9acb1c608e4f02aceda2e7fabf0b2b" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45105 } }
I20260926 02:10:10.176308 32459 raft_consensus.cc:348] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a93a1b11ee145f9bb578208730121f9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42373 } } peers { permanent_uuid: "a375b632ef6d4309be928de09a0288ff" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35361 } } peers { permanent_uuid: "9e9acb1c608e4f02aceda2e7fabf0b2b" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45105 } }
I20260926 02:10:10.176859 32459 raft_consensus.cc:374] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:10:10.177022 32459 raft_consensus.cc:729] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a93a1b11ee145f9bb578208730121f9, State: Initialized, Role: FOLLOWER
I20260926 02:10:10.177886 32459 consensus_queue.cc:260] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a93a1b11ee145f9bb578208730121f9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42373 } } peers { permanent_uuid: "a375b632ef6d4309be928de09a0288ff" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35361 } } peers { permanent_uuid: "9e9acb1c608e4f02aceda2e7fabf0b2b" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45105 } }
I20260926 02:10:10.180039 32461 tablet_bootstrap.cc:492] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P a375b632ef6d4309be928de09a0288ff: No bootstrap required, opened a new log
I20260926 02:10:10.180584 32461 ts_tablet_manager.cc:1397] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P a375b632ef6d4309be928de09a0288ff: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.012s
I20260926 02:10:10.193804 32460 ts_tablet_manager.cc:1428] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b: Time spent starting tablet: real 0.031s user 0.022s sys 0.007s
I20260926 02:10:10.195377 32152 heartbeater.cc:503] Master 127.18.159.190:45111 was elected leader, sending a full tablet report...
I20260926 02:10:10.198163 32459 ts_tablet_manager.cc:1428] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9: Time spent starting tablet: real 0.039s user 0.022s sys 0.004s
I20260926 02:10:10.201090 32466 raft_consensus.cc:482] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:10:10.201507 32466 raft_consensus.cc:504] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a93a1b11ee145f9bb578208730121f9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42373 } } peers { permanent_uuid: "a375b632ef6d4309be928de09a0288ff" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35361 } } peers { permanent_uuid: "9e9acb1c608e4f02aceda2e7fabf0b2b" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45105 } }
I20260926 02:10:10.205310 32461 raft_consensus.cc:348] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P a375b632ef6d4309be928de09a0288ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a93a1b11ee145f9bb578208730121f9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42373 } } peers { permanent_uuid: "a375b632ef6d4309be928de09a0288ff" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35361 } } peers { permanent_uuid: "9e9acb1c608e4f02aceda2e7fabf0b2b" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45105 } }
I20260926 02:10:10.206489 32461 raft_consensus.cc:374] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P a375b632ef6d4309be928de09a0288ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:10:10.206672 32461 raft_consensus.cc:729] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P a375b632ef6d4309be928de09a0288ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a375b632ef6d4309be928de09a0288ff, State: Initialized, Role: FOLLOWER
I20260926 02:10:10.207501 32466 leader_election.cc:288] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a375b632ef6d4309be928de09a0288ff (127.18.159.131:35361), 9e9acb1c608e4f02aceda2e7fabf0b2b (127.18.159.130:45105)
I20260926 02:10:10.207420 32461 consensus_queue.cc:260] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P a375b632ef6d4309be928de09a0288ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a93a1b11ee145f9bb578208730121f9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42373 } } peers { permanent_uuid: "a375b632ef6d4309be928de09a0288ff" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35361 } } peers { permanent_uuid: "9e9acb1c608e4f02aceda2e7fabf0b2b" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45105 } }
I20260926 02:10:10.218403 32461 ts_tablet_manager.cc:1428] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P a375b632ef6d4309be928de09a0288ff: Time spent starting tablet: real 0.038s user 0.022s sys 0.003s
I20260926 02:10:10.218979 32431 heartbeater.cc:503] Master 127.18.159.190:45111 was elected leader, sending a full tablet report...
I20260926 02:10:10.219776 32456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:10.205214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:32795 (local address 127.18.159.131:35361)
0926 02:10:10.205412 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:10.205418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:10.211837 (+ 6419us) server_negotiation.cc:407] Connection header received
0926 02:10:10.212354 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:10.212360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:10.212437 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:10.212500 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:10:10.213363 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.214435 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.217157 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.218508 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.218647 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:10.218719 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:10.219354 (+ 635us) server_negotiation.cc:299] Negotiation successful
0926 02:10:10.219517 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:10:10.220498 32468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:10.204638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:35361 (local address 127.18.159.129:32795)
0926 02:10:10.211669 (+ 7031us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:10.211701 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:10.212145 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:10.212732 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:10.212736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:10.212761 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:10:10.213176 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:10.213191 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.214624 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.214629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:10.216944 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:10.216960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.218948 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.218955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:10.219147 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:10.219182 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 02:10:10.219937 (+ 755us) client_negotiation.cc:240] Negotiation successful
0926 02:10:10.220088 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6883,"mutex_wait_us":1,"thread_start_us":6729,"threads_started":1}
I20260926 02:10:10.221944 32458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:10.207335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:49091 (local address 127.18.159.130:45105)
0926 02:10:10.207558 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:10.207564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:10.212108 (+ 4544us) server_negotiation.cc:407] Connection header received
0926 02:10:10.212360 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:10.212365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:10.212437 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:10.212501 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:10:10.214951 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.216046 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.219170 (+ 3124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.220408 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.220560 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:10.220649 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:10.221538 (+ 889us) server_negotiation.cc:299] Negotiation successful
0926 02:10:10.221695 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:10:10.221969 32469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:10.207297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:45105 (local address 127.18.159.129:49091)
0926 02:10:10.212005 (+ 4708us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:10.212022 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:10.212206 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:10.214283 (+ 2077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:10.214287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:10.214313 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:10:10.214710 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:10.214727 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.216279 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.216286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:10.218940 (+ 2654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:10.218959 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.221163 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.221170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:10.221343 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:10.221374 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 02:10:10.221475 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 02:10:10.221606 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4601,"mutex_wait_us":567,"thread_start_us":4571,"threads_started":1}
I20260926 02:10:10.222761 32386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e15c1f9d4e40d9b3bd6cfb4f81fd6c" candidate_uuid: "5a93a1b11ee145f9bb578208730121f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a375b632ef6d4309be928de09a0288ff" is_pre_election: true
I20260926 02:10:10.222985 32251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e15c1f9d4e40d9b3bd6cfb4f81fd6c" candidate_uuid: "5a93a1b11ee145f9bb578208730121f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9acb1c608e4f02aceda2e7fabf0b2b" is_pre_election: true
I20260926 02:10:10.223376 32386 raft_consensus.cc:2454] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P a375b632ef6d4309be928de09a0288ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a93a1b11ee145f9bb578208730121f9 in term 0.
I20260926 02:10:10.223564 32251 raft_consensus.cc:2454] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a93a1b11ee145f9bb578208730121f9 in term 0.
I20260926 02:10:10.224313 32043 leader_election.cc:302] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a93a1b11ee145f9bb578208730121f9, 9e9acb1c608e4f02aceda2e7fabf0b2b; no voters:
I20260926 02:10:10.225098 32466 raft_consensus.cc:2781] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:10:10.225271 32466 raft_consensus.cc:482] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:10:10.225373 32466 raft_consensus.cc:3037] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:10:10.227690 32466 raft_consensus.cc:504] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a93a1b11ee145f9bb578208730121f9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42373 } } peers { permanent_uuid: "a375b632ef6d4309be928de09a0288ff" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35361 } } peers { permanent_uuid: "9e9acb1c608e4f02aceda2e7fabf0b2b" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45105 } }
I20260926 02:10:10.228345 32466 leader_election.cc:288] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9 [CANDIDATE]: Term 1 election: Requested vote from peers a375b632ef6d4309be928de09a0288ff (127.18.159.131:35361), 9e9acb1c608e4f02aceda2e7fabf0b2b (127.18.159.130:45105)
I20260926 02:10:10.228807 32386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e15c1f9d4e40d9b3bd6cfb4f81fd6c" candidate_uuid: "5a93a1b11ee145f9bb578208730121f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a375b632ef6d4309be928de09a0288ff"
I20260926 02:10:10.228932 32251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e15c1f9d4e40d9b3bd6cfb4f81fd6c" candidate_uuid: "5a93a1b11ee145f9bb578208730121f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9acb1c608e4f02aceda2e7fabf0b2b"
I20260926 02:10:10.229028 32386 raft_consensus.cc:3037] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P a375b632ef6d4309be928de09a0288ff [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:10:10.229194 32251 raft_consensus.cc:3037] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:10:10.231383 32251 raft_consensus.cc:2454] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a93a1b11ee145f9bb578208730121f9 in term 1.
I20260926 02:10:10.231851 32043 leader_election.cc:302] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a93a1b11ee145f9bb578208730121f9, 9e9acb1c608e4f02aceda2e7fabf0b2b; no voters:
I20260926 02:10:10.232137 32386 raft_consensus.cc:2454] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P a375b632ef6d4309be928de09a0288ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a93a1b11ee145f9bb578208730121f9 in term 1.
I20260926 02:10:10.232203 32466 raft_consensus.cc:2781] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:10:10.232365 32466 raft_consensus.cc:686] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9 [term 1 LEADER]: Becoming Leader. State: Replica: 5a93a1b11ee145f9bb578208730121f9, State: Running, Role: LEADER
I20260926 02:10:10.232934 32466 consensus_queue.cc:237] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a93a1b11ee145f9bb578208730121f9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42373 } } peers { permanent_uuid: "a375b632ef6d4309be928de09a0288ff" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35361 } } peers { permanent_uuid: "9e9acb1c608e4f02aceda2e7fabf0b2b" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45105 } }
I20260926 02:10:10.238456 31955 catalog_manager.cc:5495] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a93a1b11ee145f9bb578208730121f9 (127.18.159.129). New cstate: current_term: 1 leader_uuid: "5a93a1b11ee145f9bb578208730121f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a93a1b11ee145f9bb578208730121f9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a375b632ef6d4309be928de09a0288ff" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e9acb1c608e4f02aceda2e7fabf0b2b" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45105 } health_report { overall_health: UNKNOWN } } }
W20260926 02:10:10.246688 32298 tablet.cc:2369] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:10:10.283347 32455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:10.277427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41798 (local address 127.18.159.129:42373)
0926 02:10:10.277695 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:10.277701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:10.277722 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:10:10.277937 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:10.277947 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:10.278034 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:10.278206 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:10:10.278629 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.279400 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.280959 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.281456 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.281612 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:10.282172 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:10:10.282699 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:10:10.282874 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:10.282931 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 02:10:10.283030 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
W20260926 02:10:10.286715 32153 tablet.cc:2369] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:10:10.292850 32386 raft_consensus.cc:1261] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P a375b632ef6d4309be928de09a0288ff [term 1 FOLLOWER]: Refusing update from remote peer 5a93a1b11ee145f9bb578208730121f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:10:10.292850 32251 raft_consensus.cc:1261] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b [term 1 FOLLOWER]: Refusing update from remote peer 5a93a1b11ee145f9bb578208730121f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:10:10.294062 32467 consensus_queue.cc:1035] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e9acb1c608e4f02aceda2e7fabf0b2b" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:10:10.294513 32466 consensus_queue.cc:1035] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a375b632ef6d4309be928de09a0288ff" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:10:10.340724 31955 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:10:10.341037 31955 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:10:10.345927 31955 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5d2f8f2cdf8421098dac41087354e7f: Sending DeleteTablet for 3 replicas of tablet e1e15c1f9d4e40d9b3bd6cfb4f81fd6c
I20260926 02:10:10.347234 32087 tablet_service.cc:1514] Processing DeleteTablet for tablet e1e15c1f9d4e40d9b3bd6cfb4f81fd6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:10:10 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:41784
I20260926 02:10:10.347280 32365 tablet_service.cc:1514] Processing DeleteTablet for tablet e1e15c1f9d4e40d9b3bd6cfb4f81fd6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:10:10 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:53992
I20260926 02:10:10.348670 32231 tablet_service.cc:1514] Processing DeleteTablet for tablet e1e15c1f9d4e40d9b3bd6cfb4f81fd6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:10:10 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:38748
I20260926 02:10:10.352097 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 31992
I20260926 02:10:10.360113 32484 tablet_replica.cc:330] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b: stopping tablet replica
I20260926 02:10:10.361383 32484 raft_consensus.cc:2229] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:10:10.361956 32484 raft_consensus.cc:2258] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:10:10.378896 32484 ts_tablet_manager.cc:1905] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:10:10.379688 32455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:10.353006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41802 (local address 127.18.159.129:42373)
0926 02:10:10.353605 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:10.353611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:10.353636 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:10:10.353712 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:10.353719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:10.353833 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:10.353957 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:10.357847 (+ 3890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.359419 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.361747 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.362820 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.363015 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:10.367618 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:10.367641 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:10.367673 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:10.367741 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:10.373419 (+ 5678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:10.374822 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:10.374837 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:10.374844 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:10.375092 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:10.375745 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:10.375753 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:10.375759 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:10.376445 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:10.378506 (+ 2061us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:10.378731 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 02:10:10.379097 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444}
I20260926 02:10:10.381032 32147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:10.390457 32483 tablet_replica.cc:330] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P a375b632ef6d4309be928de09a0288ff: stopping tablet replica
I20260926 02:10:10.390996 32483 raft_consensus.cc:2229] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P a375b632ef6d4309be928de09a0288ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:10:10.391570 32483 raft_consensus.cc:2258] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P a375b632ef6d4309be928de09a0288ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:10:10.398564 32484 ts_tablet_manager.cc:1918] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:10:10.398793 32484 log.cc:1196] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/e1e15c1f9d4e40d9b3bd6cfb4f81fd6c
I20260926 02:10:10.399365 32484 ts_tablet_manager.cc:1939] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 9e9acb1c608e4f02aceda2e7fabf0b2b: Deleting consensus metadata
I20260926 02:10:10.401656 31942 catalog_manager.cc:4841] TS 9e9acb1c608e4f02aceda2e7fabf0b2b (127.18.159.130:45105): tablet e1e15c1f9d4e40d9b3bd6cfb4f81fd6c (table test-table [id=1a6a7a9f32ce47f0b896414a17523aef]) successfully deleted
I20260926 02:10:10.401887 32483 ts_tablet_manager.cc:1905] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P a375b632ef6d4309be928de09a0288ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:10:10.406848 32483 ts_tablet_manager.cc:1918] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P a375b632ef6d4309be928de09a0288ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:10:10.406997 32483 log.cc:1196] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P a375b632ef6d4309be928de09a0288ff: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/e1e15c1f9d4e40d9b3bd6cfb4f81fd6c
I20260926 02:10:10.407526 32483 ts_tablet_manager.cc:1939] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P a375b632ef6d4309be928de09a0288ff: Deleting consensus metadata
I20260926 02:10:10.409266 31940 catalog_manager.cc:4841] TS a375b632ef6d4309be928de09a0288ff (127.18.159.131:35361): tablet e1e15c1f9d4e40d9b3bd6cfb4f81fd6c (table test-table [id=1a6a7a9f32ce47f0b896414a17523aef]) successfully deleted
I20260926 02:10:10.488735 32482 tablet_replica.cc:330] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9: stopping tablet replica
I20260926 02:10:10.488971 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 31992
I20260926 02:10:10.489351 32482 raft_consensus.cc:2229] T e1e15c1f9d4e40d9b3bd6cfb4f81fd6c P 5a93a1b11ee145f9bb578208730121f9 [term 1 LEADER]: Raft consensus shutting down.
W20260926 02:10:10.511837 31941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.159.129:42373 (error 108)
I20260926 02:10:10.511998 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 32156
I20260926 02:10:10.513871 32453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:10.513377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:42373 (local address 127.0.0.1:41808)
0926 02:10:10.513599 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:10.513707 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:42373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260926 02:10:10.514122 31941 catalog_manager.cc:4584] TS 5a93a1b11ee145f9bb578208730121f9 (127.18.159.129:42373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1e15c1f9d4e40d9b3bd6cfb4f81fd6c: Network error: Client connection negotiation failed: client connection to 127.18.159.129:42373: connect: Connection refused (error 111)
I20260926 02:10:10.526358 32458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:10.512941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38754 (local address 127.18.159.130:45105)
0926 02:10:10.513125 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:10.513131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:10.513229 (+ 98us) server_negotiation.cc:407] Connection header received
0926 02:10:10.513428 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:10.513432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:10.513496 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:10.513662 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:10.515163 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.515893 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.517288 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.517690 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.517830 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:10.520529 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:10.520543 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:10.520558 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:10.520599 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:10.523679 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:10.524443 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:10.524451 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:10.524453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:10.524604 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:10.524939 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:10.524942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:10.524944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:10.525292 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:10.525517 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:10.525770 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 02:10:10.525944 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:10:10.527104 32292 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:10.567937 32453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:10.567366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:42373 (local address 127.0.0.1:41820)
0926 02:10:10.567609 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:10.567737 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:42373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 02:10:10.569049 32453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:10.568662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:42373 (local address 127.0.0.1:41826)
0926 02:10:10.568818 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:10.568919 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:42373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 02:10:10.618165 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 32156
I20260926 02:10:10.629590 32453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:10.628966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:42373 (local address 127.0.0.1:41834)
0926 02:10:10.629242 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:10.629370 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:42373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 02:10:10.630806 32453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:10.630326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:42373 (local address 127.0.0.1:41846)
0926 02:10:10.630516 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:10.630630 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:42373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 02:10:10.636061 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 32300
I20260926 02:10:10.650172 32456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:10.636837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54000 (local address 127.18.159.131:35361)
0926 02:10:10.637006 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:10.637029 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:10.637094 (+ 65us) server_negotiation.cc:407] Connection header received
0926 02:10:10.637299 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:10.637304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:10.637377 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:10.637482 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:10.638821 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.639726 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.641164 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:10.641568 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:10.641703 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:10.644279 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:10.644294 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:10.644311 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:10.644344 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:10.647236 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:10.647980 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:10.647988 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:10.647991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:10.648127 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:10.648583 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:10.648588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:10.648592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:10.648972 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:10.649287 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:10.649599 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 02:10:10.649826 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:10:10.650884 32426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:10.711586 32453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:10.711047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:42373 (local address 127.0.0.1:41860)
0926 02:10:10.711313 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:10.711413 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:42373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260926 02:10:10.712801 32453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:10.712378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:42373 (local address 127.0.0.1:41870)
0926 02:10:10.712595 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:10.712673 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:42373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 02:10:10.742514 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 32300
I20260926 02:10:10.759320 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 31922
I20260926 02:10:10.760752 31985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:10.882148 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 31922
I20260926 02:10:10.908663 19070 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 02:10:11.245229 19070 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 02:10:11.245428 19070 mini_ranger.cc:66] Stopping Ranger...
I20260926 02:10:11.591684 19070 mini_ranger.cc:68] Stopped Ranger
2026-09-26 02:10:11.591 UTC [30974] LOG: received smart shutdown request
2026-09-26 02:10:11.595 UTC [30974] LOG: background worker "logical replication launcher" (PID 30983) exited with exit code 1
2026-09-26 02:10:11.595 UTC [30978] LOG: shutting down
2026-09-26 02:10:11.767 UTC [30974] LOG: database system is shut down
2026-09-26T02:10:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55268 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:10:11 dist-test-slave-03rn krb5kdc[32498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:10:11 dist-test-slave-03rn krb5kdc[32498](info): set up 2 sockets
Sep 26 02:10:11 dist-test-slave-03rn krb5kdc[32498](info): commencing operation
krb5kdc: starting...
W20260926 02:10:13.008900 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:10:13 dist-test-slave-03rn krb5kdc[32498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:10:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:10:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:10:13.256944 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:36139
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:36139,127.18.159.189:45957,127.18.159.188:42219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:10:13.601917 32511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:13.602217 32511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:13.602308 32511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:13.610975 32511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:10:13.611125 32511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:13.611181 32511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:13.611244 32511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:10:13.611294 32511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:10:13.623095 32511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:36139,127.18.159.189:45957,127.18.159.188:42219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:36139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:13.624562 32511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:13.625927 32511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:13.639353 32517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:13.639434 32519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:13.639459 32516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:13.640303 32511 server_base.cc:1034] running on GCE node
I20260926 02:10:13.641134 32511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:13.642709 32511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:13.643980 32511 hybrid_clock.cc:648] HybridClock initialized: now 1790388613643917 us; error 42 us; skew 500 ppm
I20260926 02:10:13.644469 32511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:13 dist-test-slave-03rn krb5kdc[32498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388613, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:13.654021 32511 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:10:13.655841 32511 webserver.cc:460] Webserver started at http://127.18.159.190:46445/ using document root <none> and password file <none>
I20260926 02:10:13.656692 32511 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:13.656813 32511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:13.657083 32511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:10:13.660399 32511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "7924772a772e4c6e8c2c36a6b912098d"
format_stamp: "Formatted at 2026-09-26 02:10:13 on dist-test-slave-03rn"
server_key: "6430d82aa3f9ae3f0c0753cc5befddae"
server_key_iv: "150fbae67f0fa910c20baf62441a0b2d"
server_key_version: "encryptionkey@0"
I20260926 02:10:13.661283 32511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "7924772a772e4c6e8c2c36a6b912098d"
format_stamp: "Formatted at 2026-09-26 02:10:13 on dist-test-slave-03rn"
server_key: "6430d82aa3f9ae3f0c0753cc5befddae"
server_key_iv: "150fbae67f0fa910c20baf62441a0b2d"
server_key_version: "encryptionkey@0"
I20260926 02:10:13.668080 32511 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 02:10:13.673564 32526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:13.675092 32511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260926 02:10:13.675257 32511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "7924772a772e4c6e8c2c36a6b912098d"
format_stamp: "Formatted at 2026-09-26 02:10:13 on dist-test-slave-03rn"
server_key: "6430d82aa3f9ae3f0c0753cc5befddae"
server_key_iv: "150fbae67f0fa910c20baf62441a0b2d"
server_key_version: "encryptionkey@0"
I20260926 02:10:13.675450 32511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:13.689383 32511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:13.693840 32511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:13.694101 32511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:13.714231 32511 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:36139
I20260926 02:10:13.714275 32577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:36139 every 8 connection(s)
I20260926 02:10:13.715795 32511 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:10:13.719991 32578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:10:13.721854 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 32511
I20260926 02:10:13.722136 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:10:13.722957 19070 external_mini_cluster.cc:1428] Setting key 4e1af20089d38415262d79e671c5f784
I20260926 02:10:13.731405 32578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.159.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.189@KRBTEST.COM" created.
Sep 26 02:10:13 dist-test-slave-03rn krb5kdc[32498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388613, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:10:13.759865 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:13.735488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:36139 (local address 127.0.0.1:49066)
0926 02:10:13.736544 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:13.736595 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:13.738871 (+ 2276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:13.739868 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:13.739875 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:13.740716 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:13.741073 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:13.741093 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:13.742924 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:13.742929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:13.744089 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:13.744104 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:13.745207 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:13.745214 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:13.745423 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:13.746168 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:13.746196 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:13.750909 (+ 4713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:13.755010 (+ 4101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:13.755029 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:13.755047 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:13.755734 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:13.756371 (+ 637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:13.756378 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:13.756383 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:13.756646 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:13.757788 (+ 1142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:13.757797 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:13.758014 (+ 217us) client_negotiation.cc:769] Sending connection context
0926 02:10:13.758245 (+ 231us) client_negotiation.cc:240] Negotiation successful
0926 02:10:13.758702 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":402,"threads_started":1}
I20260926 02:10:13.759919 32582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:13.737149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49066 (local address 127.18.159.190:36139)
0926 02:10:13.737992 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:13.738001 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:13.738054 (+ 53us) server_negotiation.cc:407] Connection header received
0926 02:10:13.739333 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:13.739340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:13.739633 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:13.739868 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:13.741299 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:13.742750 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:13.744310 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:13.744930 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:13.745173 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:13.751263 (+ 6090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:13.751273 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:13.751294 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:13.751338 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:13.754795 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:13.755912 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:13.755920 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:13.755942 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:13.756229 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:13.756829 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:13.756833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:13.756836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:13.757624 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:13.757878 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:13.758465 (+ 587us) server_negotiation.cc:299] Negotiation successful
0926 02:10:13.758942 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":627,"thread_start_us":349,"threads_started":1}
I20260926 02:10:13.763732 32578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:10:13.765151 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:13.764730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:45957 (local address 127.0.0.1:55572)
0926 02:10:13.764903 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:13.765017 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:45957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 02:10:13.765445 32528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.159.189:45957: connect: Connection refused (error 111)
I20260926 02:10:13.767186 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:13.766800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:45957 (local address 127.0.0.1:55580)
0926 02:10:13.766960 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:13.767044 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:45957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 02:10:13.767397 32578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.189:45957: Network error: Client connection negotiation failed: client connection to 127.18.159.189:45957: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.189.keytab.
Entry for principal kudu/127.18.159.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.189.keytab.
I20260926 02:10:13.783272 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.189:45957
--webserver_interface=127.18.159.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.189.keytab
--principal=kudu/127.18.159.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:36139,127.18.159.189:45957,127.18.159.188:42219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:10:13.816761 32578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } attempt: 1
I20260926 02:10:13.817978 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:13.817491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:45957 (local address 127.0.0.1:55596)
0926 02:10:13.817688 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:13.817810 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:45957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 02:10:13.819402 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:13.818816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:45957 (local address 127.0.0.1:55598)
0926 02:10:13.819088 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:13.819228 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:45957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260926 02:10:13.819646 32578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.189:45957: Network error: Client connection negotiation failed: client connection to 127.18.159.189:45957: connect: Connection refused (error 111)
I20260926 02:10:13.887943 32578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } attempt: 2
I20260926 02:10:13.889259 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:13.888750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:45957 (local address 127.0.0.1:55610)
0926 02:10:13.889020 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:13.889108 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:45957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 02:10:13.890348 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:13.889939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:45957 (local address 127.0.0.1:55620)
0926 02:10:13.890092 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:13.890192 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:45957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 02:10:13.890530 32578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.189:45957: Network error: Client connection negotiation failed: client connection to 127.18.159.189:45957: connect: Connection refused (error 111)
I20260926 02:10:13.981789 32578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } attempt: 3
I20260926 02:10:13.983054 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:13.982592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:45957 (local address 127.0.0.1:55634)
0926 02:10:13.982797 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:13.982894 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:45957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 02:10:13.984258 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:13.983799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:45957 (local address 127.0.0.1:55648)
0926 02:10:13.983972 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:13.984057 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:45957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 02:10:13.984427 32578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.189:45957: Network error: Client connection negotiation failed: client connection to 127.18.159.189:45957: connect: Connection refused (error 111)
I20260926 02:10:14.127692 32578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } attempt: 4
I20260926 02:10:14.129102 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.128471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:45957 (local address 127.0.0.1:55662)
0926 02:10:14.128830 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.128948 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:45957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
I20260926 02:10:14.130206 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.129794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:45957 (local address 127.0.0.1:55670)
0926 02:10:14.129973 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.130068 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:45957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 02:10:14.130455 32578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.189:45957: Network error: Client connection negotiation failed: client connection to 127.18.159.189:45957: connect: Connection refused (error 111)
W20260926 02:10:14.141018 32585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:14.141348 32585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:14.141520 32585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:14.149180 32585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:10:14.149307 32585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:14.149415 32585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:14.149503 32585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:10:14.149582 32585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:10:14.160310 32585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:36139,127.18.159.189:45957,127.18.159.188:42219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.189.keytab
--principal=kudu/127.18.159.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.189:45957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.159.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:14.161775 32585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:14.163183 32585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:14.175784 32591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:14.175782 32590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:14.176365 32593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:14.176860 32585 server_base.cc:1034] running on GCE node
I20260926 02:10:14.177737 32585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:14.179304 32585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:14.180557 32585 hybrid_clock.cc:648] HybridClock initialized: now 1790388614180474 us; error 77 us; skew 500 ppm
I20260926 02:10:14.181010 32585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:14 dist-test-slave-03rn krb5kdc[32498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388614, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:14.184460 32585 init.cc:377] Logged in from keytab as kudu/127.18.159.189@KRBTEST.COM (short username kudu)
I20260926 02:10:14.186381 32585 webserver.cc:460] Webserver started at http://127.18.159.189:39931/ using document root <none> and password file <none>
I20260926 02:10:14.187183 32585 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:14.187335 32585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:14.187701 32585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:10:14.191119 32585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-1/data/instance:
uuid: "078c6c5cd88c4a7ca1edeb66f668f88b"
format_stamp: "Formatted at 2026-09-26 02:10:14 on dist-test-slave-03rn"
server_key: "a04ee8f8c617d01d6bf7a2d82ef44877"
server_key_iv: "93ded82c3c6df22d50a23510f8bcf38b"
server_key_version: "encryptionkey@0"
I20260926 02:10:14.192062 32585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-1/wal/instance:
uuid: "078c6c5cd88c4a7ca1edeb66f668f88b"
format_stamp: "Formatted at 2026-09-26 02:10:14 on dist-test-slave-03rn"
server_key: "a04ee8f8c617d01d6bf7a2d82ef44877"
server_key_iv: "93ded82c3c6df22d50a23510f8bcf38b"
server_key_version: "encryptionkey@0"
I20260926 02:10:14.198414 32585 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:10:14.202415 32600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:14.204041 32585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:10:14.204311 32585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-1/wal
uuid: "078c6c5cd88c4a7ca1edeb66f668f88b"
format_stamp: "Formatted at 2026-09-26 02:10:14 on dist-test-slave-03rn"
server_key: "a04ee8f8c617d01d6bf7a2d82ef44877"
server_key_iv: "93ded82c3c6df22d50a23510f8bcf38b"
server_key_version: "encryptionkey@0"
I20260926 02:10:14.204552 32585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:14.228539 32585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:14.233525 32585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:14.233867 32585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:14.254225 32585 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.189:45957
I20260926 02:10:14.254257 32651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.189:45957 every 8 connection(s)
I20260926 02:10:14.255846 32585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-1/data/info.pb
I20260926 02:10:14.257704 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 32585
I20260926 02:10:14.257905 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-1/wal/instance
I20260926 02:10:14.258374 19070 external_mini_cluster.cc:1428] Setting key 8a64c2d2ec3dfa3741dd88f204de625d
I20260926 02:10:14.260342 32652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:10:14.268754 32652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.159.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.188@KRBTEST.COM" created.
Sep 26 02:10:14 dist-test-slave-03rn krb5kdc[32498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388614, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.189@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:10:14.296494 32655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.272718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49076 (local address 127.18.159.190:36139)
0926 02:10:14.273288 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:14.273295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:14.274616 (+ 1321us) server_negotiation.cc:407] Connection header received
0926 02:10:14.278036 (+ 3420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:14.278041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:14.278125 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:14.278236 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:14.281022 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.281891 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.283773 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.284339 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.284494 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.288901 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:14.288906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:14.288910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:14.288961 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:14.292425 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.293457 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.293466 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.293472 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.293640 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.294173 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.294181 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.294187 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.294620 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:14.294787 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:14.295642 (+ 855us) server_negotiation.cc:299] Negotiation successful
0926 02:10:14.295930 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":276,"threads_started":1}
I20260926 02:10:14.297247 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.273438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:36139 (local address 127.0.0.1:49076)
0926 02:10:14.274391 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.274453 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:14.277560 (+ 3107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:14.278610 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:14.278627 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:14.279649 (+ 1022us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:14.280701 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:14.280738 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.282112 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.282116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:14.283596 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:14.283613 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.284617 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.284627 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:14.284889 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.285680 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:14.285709 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:14.288572 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:14.292704 (+ 4132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:14.292713 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:14.292730 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:14.293281 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:14.293788 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:14.293792 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:14.293795 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:14.294026 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:14.294774 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:14.294782 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:14.295224 (+ 442us) client_negotiation.cc:769] Sending connection context
0926 02:10:14.295500 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 02:10:14.295939 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":367,"threads_started":1}
I20260926 02:10:14.299408 32652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:10:14 dist-test-slave-03rn krb5kdc[32498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388614, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.189@KRBTEST.COM for kudu/127.18.159.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.188.keytab.
Entry for principal kudu/127.18.159.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.188.keytab.
I20260926 02:10:14.315740 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.188:42219
--webserver_interface=127.18.159.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.188.keytab
--principal=kudu/127.18.159.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:36139,127.18.159.189:45957,127.18.159.188:42219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:10:14.316752 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.300355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:45957 (local address 127.0.0.1:55678)
0926 02:10:14.300580 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.300600 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:14.300739 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:14.301512 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:14.301517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:14.302006 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:14.302145 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:14.302153 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.303573 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.303581 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:14.304499 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:14.304511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.305360 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.305366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:14.305625 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.306221 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:14.306240 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:14.309618 (+ 3378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:14.313684 (+ 4066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:14.313689 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:14.313692 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:14.314095 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:14.314683 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:14.314692 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:14.314695 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:14.314853 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:14.315702 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:14.315706 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:14.315827 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 02:10:14.315979 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 02:10:14.316344 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 02:10:14.316890 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.300457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55678 (local address 127.18.159.189:45957)
0926 02:10:14.301066 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:14.301072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:14.301099 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:10:14.301145 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:14.301150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:14.301344 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:14.301530 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:14.302307 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.303430 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.304634 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.305208 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.305346 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.309878 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:14.309893 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:14.309907 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:14.309946 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:14.313506 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.314336 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.314344 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.314347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.314562 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.314997 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.315002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.315006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.315536 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:14.315782 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:14.316287 (+ 505us) server_negotiation.cc:299] Negotiation successful
0926 02:10:14.316523 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":296,"threads_started":1}
I20260926 02:10:14.319598 32652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:10:14.320983 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.320353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:42219 (local address 127.0.0.1:49940)
0926 02:10:14.320575 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.320728 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 02:10:14.321280 32602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
I20260926 02:10:14.323109 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.322616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:42219 (local address 127.0.0.1:49954)
0926 02:10:14.322828 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.322940 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 02:10:14.323352 32652 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:42219: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
I20260926 02:10:14.373556 32652 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } attempt: 1
I20260926 02:10:14.374938 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.374391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:42219 (local address 127.0.0.1:49966)
0926 02:10:14.374632 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.374764 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 02:10:14.376500 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.375991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:42219 (local address 127.0.0.1:49980)
0926 02:10:14.376217 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.376320 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 02:10:14.376761 32652 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:42219: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
I20260926 02:10:14.429783 32578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } attempt: 5
Sep 26 02:10:14 dist-test-slave-03rn krb5kdc[32498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388613, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for kudu/127.18.159.189@KRBTEST.COM
I20260926 02:10:14.444401 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.430856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55680 (local address 127.18.159.189:45957)
0926 02:10:14.431048 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:14.431053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:14.431098 (+ 45us) server_negotiation.cc:407] Connection header received
0926 02:10:14.431160 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:14.431164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:14.431243 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:14.431326 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:14.432708 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.433431 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.434688 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.435151 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.435307 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.438679 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:14.438686 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:14.438690 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:14.438729 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:14.441530 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.442338 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.442343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.442345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.442478 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.442887 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.442892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.442895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.443205 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:14.443371 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:14.443778 (+ 407us) server_negotiation.cc:299] Negotiation successful
0926 02:10:14.444015 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:10:14.444432 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.430610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:45957 (local address 127.0.0.1:55680)
0926 02:10:14.430869 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.430892 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:14.431030 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:14.431402 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:14.431409 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:14.432022 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:14.432463 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:14.432482 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.433593 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.433599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:14.434507 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:14.434521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.435334 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.435341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:14.435546 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.436248 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:14.436268 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:14.438425 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:14.441730 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:14.441737 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:14.441741 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:14.442182 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:14.442608 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:14.442613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:14.442617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:14.442770 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:14.443331 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:14.443336 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:14.443512 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 02:10:14.443672 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 02:10:14.444034 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 02:10:14.445000 32652 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } attempt: 2
I20260926 02:10:14.445644 32578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:10:14.446048 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.445581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:42219 (local address 127.0.0.1:49994)
0926 02:10:14.445776 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.445877 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 02:10:14.446825 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.446315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:42219 (local address 127.0.0.1:50006)
0926 02:10:14.446539 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.446648 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260926 02:10:14.447124 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.446733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:42219 (local address 127.0.0.1:50014)
0926 02:10:14.446882 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.446965 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 02:10:14.447324 32652 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:42219: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
I20260926 02:10:14.447800 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.447437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:42219 (local address 127.0.0.1:50024)
0926 02:10:14.447608 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.447687 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260926 02:10:14.448086 32578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:42219: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
I20260926 02:10:14.499400 32578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } attempt: 1
I20260926 02:10:14.500715 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.500254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:42219 (local address 127.0.0.1:50032)
0926 02:10:14.500446 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.500542 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 02:10:14.501809 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.501463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:42219 (local address 127.0.0.1:50042)
0926 02:10:14.501621 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.501701 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 02:10:14.502038 32578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:42219: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
I20260926 02:10:14.538621 32652 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } attempt: 3
I20260926 02:10:14.539968 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.539368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:42219 (local address 127.0.0.1:50052)
0926 02:10:14.539624 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.539745 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 02:10:14.541445 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.540844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:42219 (local address 127.0.0.1:50066)
0926 02:10:14.541145 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.541270 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260926 02:10:14.541770 32652 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:42219: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
I20260926 02:10:14.570320 32578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } attempt: 2
I20260926 02:10:14.571627 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.571108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:42219 (local address 127.0.0.1:50074)
0926 02:10:14.571352 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.571458 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 02:10:14.572938 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.572519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:42219 (local address 127.0.0.1:50086)
0926 02:10:14.572707 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.572808 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 02:10:14.573175 32578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:42219: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
W20260926 02:10:14.670356 32660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:14.670746 32660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:14.670899 32660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:14.678479 32660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:10:14.678668 32660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:14.678766 32660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:14.678858 32660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:10:14.678936 32660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:10:14.679446 32578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } attempt: 3
I20260926 02:10:14.680779 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.680260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:42219 (local address 127.0.0.1:50096)
0926 02:10:14.680459 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.680571 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 02:10:14.682000 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.681602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:42219 (local address 127.0.0.1:50098)
0926 02:10:14.681791 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.681888 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260926 02:10:14.682216 32578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:42219: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
I20260926 02:10:14.685017 32652 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } attempt: 4
I20260926 02:10:14.686226 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.685633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:42219 (local address 127.0.0.1:50100)
0926 02:10:14.685888 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.686005 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 02:10:14.687273 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.686892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:42219 (local address 127.0.0.1:50112)
0926 02:10:14.687045 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.687141 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 02:10:14.687487 32652 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:42219: Network error: Client connection negotiation failed: client connection to 127.18.159.188:42219: connect: Connection refused (error 111)
I20260926 02:10:14.690917 32660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:36139,127.18.159.189:45957,127.18.159.188:42219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.188.keytab
--principal=kudu/127.18.159.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.188:42219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.159.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:14.692412 32660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:14.693828 32660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:14.706928 32666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:14.706902 32665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:14.707118 32668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:14.707897 32660 server_base.cc:1034] running on GCE node
I20260926 02:10:14.708953 32660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:14.710635 32660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:14.711895 32660 hybrid_clock.cc:648] HybridClock initialized: now 1790388614711832 us; error 62 us; skew 500 ppm
I20260926 02:10:14.712356 32660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:14 dist-test-slave-03rn krb5kdc[32498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388614, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:14.716413 32660 init.cc:377] Logged in from keytab as kudu/127.18.159.188@KRBTEST.COM (short username kudu)
I20260926 02:10:14.718315 32660 webserver.cc:460] Webserver started at http://127.18.159.188:44659/ using document root <none> and password file <none>
I20260926 02:10:14.719106 32660 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:14.719244 32660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:14.719596 32660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:10:14.722841 32660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-2/data/instance:
uuid: "73258d93a2bd49b9b4eb548c807487c3"
format_stamp: "Formatted at 2026-09-26 02:10:14 on dist-test-slave-03rn"
server_key: "542cbcc6724a2cb09bacd0bcc79f2742"
server_key_iv: "61a9a34d6e739026dc84bf2c303e8035"
server_key_version: "encryptionkey@0"
I20260926 02:10:14.723750 32660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-2/wal/instance:
uuid: "73258d93a2bd49b9b4eb548c807487c3"
format_stamp: "Formatted at 2026-09-26 02:10:14 on dist-test-slave-03rn"
server_key: "542cbcc6724a2cb09bacd0bcc79f2742"
server_key_iv: "61a9a34d6e739026dc84bf2c303e8035"
server_key_version: "encryptionkey@0"
I20260926 02:10:14.729496 32660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 02:10:14.733541 32675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:14.735118 32660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260926 02:10:14.735347 32660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-2/wal
uuid: "73258d93a2bd49b9b4eb548c807487c3"
format_stamp: "Formatted at 2026-09-26 02:10:14 on dist-test-slave-03rn"
server_key: "542cbcc6724a2cb09bacd0bcc79f2742"
server_key_iv: "61a9a34d6e739026dc84bf2c303e8035"
server_key_version: "encryptionkey@0"
I20260926 02:10:14.735621 32660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:14.761245 32660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:14.765899 32660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:14.766361 32660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:14.785709 32660 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.188:42219
I20260926 02:10:14.785741 32726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.188:42219 every 8 connection(s)
I20260926 02:10:14.787184 32660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-2/data/info.pb
I20260926 02:10:14.791071 32727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:10:14.791816 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 32660
I20260926 02:10:14.792025 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388402166281-19070-0/minicluster-data/master-2/wal/instance
I20260926 02:10:14.792578 19070 external_mini_cluster.cc:1428] Setting key 7e0696ec5860069ab186fa96edb50d68
I20260926 02:10:14.800688 32727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:10:14 dist-test-slave-03rn krb5kdc[32498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.188@KRBTEST.COM
Sep 26 02:10:14 dist-test-slave-03rn krb5kdc[32498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388614, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.188@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:10:14.818910 32730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.794546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50126 (local address 127.18.159.188:42219)
0926 02:10:14.795380 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:14.795392 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:14.795438 (+ 46us) server_negotiation.cc:407] Connection header received
0926 02:10:14.797289 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:14.797316 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:14.797755 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:14.798129 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:14.800025 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.801474 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.803452 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.804690 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.804954 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.810305 (+ 5351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:14.810314 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:14.810326 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:14.810372 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:14.814125 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.815074 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.815083 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.815096 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.815279 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.815658 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.815666 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.815670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.816325 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:14.816630 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:14.817088 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 02:10:14.817544 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":401,"threads_started":1}
I20260926 02:10:14.820844 32732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.803754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:36139 (local address 127.0.0.1:49084)
0926 02:10:14.804648 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.804690 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:14.804919 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:14.805410 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:14.805415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:14.806215 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:14.806589 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:14.806606 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.808216 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.808220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:14.809745 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:14.809758 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.810673 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.810679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:14.810893 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.811672 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:14.811701 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:14.814634 (+ 2933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:14.817972 (+ 3338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:14.817982 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:14.817992 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:14.818583 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:14.819015 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:14.819020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:14.819023 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:14.819194 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:14.819778 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:14.819784 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:14.819979 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 02:10:14.820235 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 02:10:14.820558 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":523,"threads_started":1}
I20260926 02:10:14.821055 32731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.803742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49084 (local address 127.18.159.190:36139)
0926 02:10:14.804328 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:14.804334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:14.804786 (+ 452us) server_negotiation.cc:407] Connection header received
0926 02:10:14.805137 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:14.805144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:14.805228 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:14.805343 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:14.806776 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.808006 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.809913 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.810490 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.810597 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.814840 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:14.814845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:14.814848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:14.814884 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:14.817838 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.818741 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.818746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.818748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.818894 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.819343 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.819355 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.819357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.819651 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:14.819767 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:14.820384 (+ 617us) server_negotiation.cc:299] Negotiation successful
0926 02:10:14.820639 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":261,"threads_started":1}
I20260926 02:10:14.822499 32727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:10:14 dist-test-slave-03rn krb5kdc[32498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388614, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.188@KRBTEST.COM for kudu/127.18.159.189@KRBTEST.COM
Sep 26 02:10:14 dist-test-slave-03rn krb5kdc[32498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:10:14.838325 32732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.823334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:45957 (local address 127.0.0.1:55684)
0926 02:10:14.823520 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.823539 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:14.823646 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:14.823950 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:14.823956 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:14.824477 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:14.824697 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:14.824712 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.826048 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.826052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:14.827422 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:14.827438 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.828277 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.828282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:14.828440 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.829081 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:14.829104 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:14.831679 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:14.835585 (+ 3906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:14.835589 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:14.835592 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:14.836010 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:14.836524 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:14.836528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:14.836531 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:14.836682 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:14.837320 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:14.837325 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:14.837564 (+ 239us) client_negotiation.cc:769] Sending connection context
0926 02:10:14.837685 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 02:10:14.838017 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 02:10:14.838493 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.823308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55684 (local address 127.18.159.189:45957)
0926 02:10:14.823510 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:14.823517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:14.823584 (+ 67us) server_negotiation.cc:407] Connection header received
0926 02:10:14.823751 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:14.823755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:14.823812 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:14.823891 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:14.824864 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.825897 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.827584 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.828099 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.828262 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.831903 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:14.831908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:14.831912 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:14.831973 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:14.835415 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.836146 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.836182 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.836188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.836364 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.836871 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.836876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.836880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.837177 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:14.837284 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:14.837831 (+ 547us) server_negotiation.cc:299] Negotiation successful
0926 02:10:14.838098 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:10:14.839668 32727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:10:14.839978 32731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.822697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.18.159.190:36139)
0926 02:10:14.822883 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:14.822888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:14.822909 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:10:14.823004 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:14.823008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:14.823107 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:14.824243 (+ 1136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:14.825178 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.826161 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.827914 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.828560 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.828654 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.833366 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:14.833374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:14.833379 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:14.833424 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:14.836680 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.837517 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.837521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.837523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.837652 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.838166 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.838171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.838174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.838566 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:14.838856 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:14.839350 (+ 494us) server_negotiation.cc:299] Negotiation successful
0926 02:10:14.839646 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 02:10:14 dist-test-slave-03rn krb5kdc[32498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388614, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.188@KRBTEST.COM for kudu/127.18.159.188@KRBTEST.COM
Sep 26 02:10:14 dist-test-slave-03rn krb5kdc[32498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.189@KRBTEST.COM
I20260926 02:10:14.853449 32730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.840511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.18.159.188:42219)
0926 02:10:14.840678 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:14.840684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:14.840771 (+ 87us) server_negotiation.cc:407] Connection header received
0926 02:10:14.840954 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:14.840959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:14.841022 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:14.841122 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:14.841926 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.842767 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.844214 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.844788 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.844879 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.848355 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:14.848360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:14.848363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:14.848390 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:14.850927 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.851555 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.851563 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.851567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.851684 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.852030 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.852036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.852039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.852368 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:14.852514 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:14.852842 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 02:10:14.853065 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 02:10:14.853572 32732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.840467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:42219 (local address 127.0.0.1:50140)
0926 02:10:14.840708 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.840725 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:14.840835 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:14.841129 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:14.841133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:14.841585 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:14.841772 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:14.841785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.842908 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.842916 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:14.844047 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:14.844059 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.844954 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.844961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:14.845171 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.845791 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:14.845814 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:14.848135 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:14.851059 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:14.851067 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:14.851072 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:14.851436 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:14.851787 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:14.851793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:14.851797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:14.851925 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:14.852474 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:14.852479 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:14.852628 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 02:10:14.852771 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 02:10:14.853215 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":139}
I20260926 02:10:14.856555 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.841702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55686 (local address 127.18.159.189:45957)
0926 02:10:14.841876 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:14.841881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:14.841900 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:10:14.842034 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:14.842041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:14.842115 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:14.842216 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:14.843570 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.844442 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.846096 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.846510 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.846610 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.850929 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:14.850934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:14.850938 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:14.850971 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:14.853852 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.854582 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.854588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.854592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.854768 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.855190 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.855194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.855196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.855459 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:14.855676 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:14.856008 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 02:10:14.856272 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 02:10:14.859427 32727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3: Bootstrap starting.
I20260926 02:10:14.859469 32578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } attempt: 4
I20260926 02:10:14.862703 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:10:14.863727 32727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3: Neither blocks nor log segments found. Creating new log.
I20260926 02:10:14.865298 32727 log.cc:824] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:10:14 dist-test-slave-03rn krb5kdc[32498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388613, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for kudu/127.18.159.188@KRBTEST.COM
I20260926 02:10:14.868927 32727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3: No bootstrap required, opened a new log
I20260926 02:10:14.877646 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.859989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:42219 (local address 127.0.0.1:50152)
0926 02:10:14.860274 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.860295 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:14.860399 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:14.860747 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:14.860752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:14.861150 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:14.861309 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:14.861318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.862678 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.862684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:14.863726 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:14.863740 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.864639 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.864645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:14.864836 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.865477 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:14.865503 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:14.868287 (+ 2784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:14.873415 (+ 5128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:14.873425 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:14.873432 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:14.874196 (+ 764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:14.874838 (+ 642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:14.874847 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:14.874854 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:14.875092 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:14.876098 (+ 1006us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:14.876106 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:14.876383 (+ 277us) client_negotiation.cc:769] Sending connection context
0926 02:10:14.876590 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 02:10:14.877042 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260926 02:10:14.879205 32730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.860092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50152 (local address 127.18.159.188:42219)
0926 02:10:14.860319 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:14.860324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:14.860344 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:10:14.860553 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:14.860557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:14.860616 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:14.860715 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:14.861453 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.862450 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.863928 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.864485 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.864616 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.868542 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:14.868550 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:14.868555 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:14.868595 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:14.873188 (+ 4593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.874430 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.874438 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.874441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.874662 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.875254 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.875262 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.875268 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.875801 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:14.876013 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:14.878280 (+ 2267us) server_negotiation.cc:299] Negotiation successful
0926 02:10:14.878624 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 02:10:14.884537 32727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7924772a772e4c6e8c2c36a6b912098d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } } peers { permanent_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } } peers { permanent_uuid: "73258d93a2bd49b9b4eb548c807487c3" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } }
I20260926 02:10:14.885363 32727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:10:14.885566 32727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73258d93a2bd49b9b4eb548c807487c3, State: Initialized, Role: FOLLOWER
I20260926 02:10:14.886500 32727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7924772a772e4c6e8c2c36a6b912098d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } } peers { permanent_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } } peers { permanent_uuid: "73258d93a2bd49b9b4eb548c807487c3" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } }
I20260926 02:10:14.893596 32742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7924772a772e4c6e8c2c36a6b912098d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } } peers { permanent_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } } peers { permanent_uuid: "73258d93a2bd49b9b4eb548c807487c3" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } } }
I20260926 02:10:14.895269 32578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d: Bootstrap starting.
I20260926 02:10:14.896143 32742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:10:14.896006 32727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:10:14.900141 32578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d: Neither blocks nor log segments found. Creating new log.
I20260926 02:10:14.902021 32731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.869727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49110 (local address 127.18.159.190:36139)
0926 02:10:14.869955 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:14.869961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:14.880347 (+ 10386us) server_negotiation.cc:407] Connection header received
0926 02:10:14.880750 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:14.880759 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:14.880863 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:14.880961 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:14.884073 (+ 3112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.885345 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.887988 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.888688 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.888817 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.892658 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:14.892667 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:14.892675 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:14.892741 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:14.896745 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.898003 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.898009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.898015 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.898490 (+ 475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.899043 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.899050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.899054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.899476 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:14.899762 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:14.901328 (+ 1566us) server_negotiation.cc:299] Negotiation successful
0926 02:10:14.901598 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 02:10:14.902385 32578 log.cc:824] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:10:14.903573 32730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.873088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50166 (local address 127.18.159.188:42219)
0926 02:10:14.879709 (+ 6621us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:14.879722 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:14.885093 (+ 5371us) server_negotiation.cc:407] Connection header received
0926 02:10:14.885544 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:14.885555 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:14.885641 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:14.885715 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:14.888286 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.889415 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.891662 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.892394 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.892519 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.896120 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:14.896134 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:14.896140 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:14.896207 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:14.900181 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.901087 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.901092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.901096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.901267 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.901724 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.901731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.901737 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.902081 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:14.902317 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:14.902953 (+ 636us) server_negotiation.cc:299] Negotiation successful
0926 02:10:14.903207 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6466,"thread_start_us":14887,"threads_started":1}
I20260926 02:10:14.914870 32578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d: No bootstrap required, opened a new log
I20260926 02:10:14.918622 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.869869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55688 (local address 127.18.159.189:45957)
0926 02:10:14.870065 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:14.870072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:14.880342 (+ 10270us) server_negotiation.cc:407] Connection header received
0926 02:10:14.880738 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:14.880745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:14.880834 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:14.880904 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:14.885612 (+ 4708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.886666 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.905185 (+ 18519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.905905 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.906067 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.909914 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:14.909920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:14.909923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:14.909973 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:14.913728 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.915070 (+ 1342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.915081 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.915087 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.915305 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:14.915902 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:14.915911 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:14.915917 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:14.916548 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:14.916897 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:14.917717 (+ 820us) server_negotiation.cc:299] Negotiation successful
0926 02:10:14.918097 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:10:14.912998 32660 master_runner.cc:186] Error getting master registration for 127.18.159.190:36139: OK, instance_id { permanent_uuid: "7924772a772e4c6e8c2c36a6b912098d" instance_seqno: 1790388613708382 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:10:14.924440 32660 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:10:14.927773 32578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7924772a772e4c6e8c2c36a6b912098d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } } peers { permanent_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } } peers { permanent_uuid: "73258d93a2bd49b9b4eb548c807487c3" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } }
I20260926 02:10:14.928117 32578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:10:14.928292 32578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7924772a772e4c6e8c2c36a6b912098d, State: Initialized, Role: FOLLOWER
I20260926 02:10:14.928884 32578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7924772a772e4c6e8c2c36a6b912098d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } } peers { permanent_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } } peers { permanent_uuid: "73258d93a2bd49b9b4eb548c807487c3" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } }
I20260926 02:10:14.929857 32756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7924772a772e4c6e8c2c36a6b912098d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } } peers { permanent_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } } peers { permanent_uuid: "73258d93a2bd49b9b4eb548c807487c3" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } } }
I20260926 02:10:14.930248 32756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [sys.catalog]: This master's current role is: FOLLOWER
W20260926 02:10:14.930394 32754 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:10:14.930543 32754 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:10:14.931149 32578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [sys.catalog]: configured and running, proceeding with master startup.
W20260926 02:10:14.944977 32767 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:10:14.945263 32767 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 02:10:14.947089 32532 tablet.cc:2369] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:10:14.943342 32511 master_runner.cc:186] Error getting master registration for 127.18.159.189:45957: OK, instance_id { permanent_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" instance_seqno: 1790388614248181 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:10:14.955166 32511 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:10:14.986711 32652 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } attempt: 5
Sep 26 02:10:14 dist-test-slave-03rn krb5kdc[32498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388614, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.189@KRBTEST.COM for kudu/127.18.159.188@KRBTEST.COM
I20260926 02:10:15.008602 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.987448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:42219 (local address 127.0.0.1:50174)
0926 02:10:14.987880 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:14.987902 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:14.988052 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:14.989150 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:14.989157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:14.989760 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:14.990011 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:14.990029 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.991691 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.991699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:14.993221 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:14.993242 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.994444 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.994451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:14.994697 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.995456 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:14.995482 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:14.998674 (+ 3192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:15.005143 (+ 6469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:15.005151 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:15.005156 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:15.005834 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:15.006359 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:15.006365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:15.006369 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:15.006557 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:15.007316 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:15.007322 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:15.007509 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 02:10:15.007773 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 02:10:15.008138 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287}
I20260926 02:10:15.008836 32730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:14.988319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50174 (local address 127.18.159.188:42219)
0926 02:10:14.988538 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:14.988545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:14.988565 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:10:14.988628 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:14.988634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:14.988723 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:14.988833 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:14.990220 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.991481 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.993417 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:14.994247 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:14.994398 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:14.998970 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:14.998983 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:14.998989 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:14.999044 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:15.004934 (+ 5890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:15.006014 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:15.006020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:15.006023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:15.006217 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:15.006718 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:15.006726 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:15.006732 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:15.007175 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:15.007365 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:15.007963 (+ 598us) server_negotiation.cc:299] Negotiation successful
0926 02:10:15.008305 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 02:10:15.015645 32652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b: Bootstrap starting.
W20260926 02:10:15.018714 32681 tablet.cc:2369] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:10:15.020488 32652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b: Neither blocks nor log segments found. Creating new log.
I20260926 02:10:15.022130 32652 log.cc:824] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b: Log is configured to *not* fsync() on all Append() calls
I20260926 02:10:15.026083 32652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b: No bootstrap required, opened a new log
I20260926 02:10:15.039791 32652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7924772a772e4c6e8c2c36a6b912098d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } } peers { permanent_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } } peers { permanent_uuid: "73258d93a2bd49b9b4eb548c807487c3" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } }
I20260926 02:10:15.040380 32652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:10:15.040566 32652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078c6c5cd88c4a7ca1edeb66f668f88b, State: Initialized, Role: FOLLOWER
I20260926 02:10:15.041358 32652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7924772a772e4c6e8c2c36a6b912098d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } } peers { permanent_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } } peers { permanent_uuid: "73258d93a2bd49b9b4eb548c807487c3" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } }
I20260926 02:10:15.042512 302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7924772a772e4c6e8c2c36a6b912098d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } } peers { permanent_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } } peers { permanent_uuid: "73258d93a2bd49b9b4eb548c807487c3" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } } }
I20260926 02:10:15.043018 302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:10:15.044096 32652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:10:15.066905 32585 master_runner.cc:263] No leader master found from master 078c6c5cd88c4a7ca1edeb66f668f88b
I20260926 02:10:15.067107 32585 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:10:15.075361 313 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:10:15.075609 313 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:10:15.103891 32756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:10:15.104421 32756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7924772a772e4c6e8c2c36a6b912098d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } } peers { permanent_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } } peers { permanent_uuid: "73258d93a2bd49b9b4eb548c807487c3" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } }
I20260926 02:10:15.106200 32756 leader_election.cc:288] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 078c6c5cd88c4a7ca1edeb66f668f88b (127.18.159.189:45957), 73258d93a2bd49b9b4eb548c807487c3 (127.18.159.188:42219)
I20260926 02:10:15.107846 32627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7924772a772e4c6e8c2c36a6b912098d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" is_pre_election: true
I20260926 02:10:15.108520 32627 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7924772a772e4c6e8c2c36a6b912098d in term 0.
I20260926 02:10:15.109793 32702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7924772a772e4c6e8c2c36a6b912098d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73258d93a2bd49b9b4eb548c807487c3" is_pre_election: true
I20260926 02:10:15.110417 32702 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7924772a772e4c6e8c2c36a6b912098d in term 0.
I20260926 02:10:15.111215 32528 leader_election.cc:302] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73258d93a2bd49b9b4eb548c807487c3, 7924772a772e4c6e8c2c36a6b912098d; no voters:
I20260926 02:10:15.116345 32756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:10:15.116564 32756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:10:15.116671 32756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:10:15.120416 32756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7924772a772e4c6e8c2c36a6b912098d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } } peers { permanent_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } } peers { permanent_uuid: "73258d93a2bd49b9b4eb548c807487c3" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } }
I20260926 02:10:15.121922 32627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7924772a772e4c6e8c2c36a6b912098d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b"
I20260926 02:10:15.122181 32627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:10:15.125887 32627 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7924772a772e4c6e8c2c36a6b912098d in term 1.
I20260926 02:10:15.126613 32528 leader_election.cc:302] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 078c6c5cd88c4a7ca1edeb66f668f88b, 7924772a772e4c6e8c2c36a6b912098d; no voters:
I20260926 02:10:15.129231 32756 leader_election.cc:288] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [CANDIDATE]: Term 1 election: Requested vote from peers 078c6c5cd88c4a7ca1edeb66f668f88b (127.18.159.189:45957), 73258d93a2bd49b9b4eb548c807487c3 (127.18.159.188:42219)
I20260926 02:10:15.129859 32756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:10:15.130332 32756 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [term 1 LEADER]: Becoming Leader. State: Replica: 7924772a772e4c6e8c2c36a6b912098d, State: Running, Role: LEADER
I20260926 02:10:15.131830 32756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7924772a772e4c6e8c2c36a6b912098d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } } peers { permanent_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } } peers { permanent_uuid: "73258d93a2bd49b9b4eb548c807487c3" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } }
I20260926 02:10:15.146528 32702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7924772a772e4c6e8c2c36a6b912098d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73258d93a2bd49b9b4eb548c807487c3"
I20260926 02:10:15.146824 32702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:10:15.150395 32702 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7924772a772e4c6e8c2c36a6b912098d in term 1.
I20260926 02:10:15.154064 315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7924772a772e4c6e8c2c36a6b912098d. Latest consensus state: current_term: 1 leader_uuid: "7924772a772e4c6e8c2c36a6b912098d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7924772a772e4c6e8c2c36a6b912098d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } } peers { permanent_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } } peers { permanent_uuid: "73258d93a2bd49b9b4eb548c807487c3" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } } }
I20260926 02:10:15.154663 315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [sys.catalog]: This master's current role is: LEADER
I20260926 02:10:15.166443 317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:10:15.169021 317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:10:15.193888 32702 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3 [term 1 FOLLOWER]: Refusing update from remote peer 7924772a772e4c6e8c2c36a6b912098d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:10:15.195209 32627 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b [term 1 FOLLOWER]: Refusing update from remote peer 7924772a772e4c6e8c2c36a6b912098d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:10:15.196586 32756 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [LEADER]: Connected to new peer: Peer: permanent_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:10:15.202456 32756 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [LEADER]: Connected to new peer: Peer: permanent_uuid: "73258d93a2bd49b9b4eb548c807487c3" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 02:10:15.236745 32606 tablet.cc:2369] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:10:15.247687 32742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7924772a772e4c6e8c2c36a6b912098d. Latest consensus state: current_term: 1 leader_uuid: "7924772a772e4c6e8c2c36a6b912098d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7924772a772e4c6e8c2c36a6b912098d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } } peers { permanent_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } } peers { permanent_uuid: "73258d93a2bd49b9b4eb548c807487c3" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } } }
I20260926 02:10:15.248126 32742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:10:15.250425 318 mvcc.cc:205] Tried to move back new op lower bound from 7333431767819210752 to 7333431767593508864. Current Snapshot: MvccSnapshot[applied={T|T < 7333431767819210752}]
I20260926 02:10:15.252313 32756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7924772a772e4c6e8c2c36a6b912098d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7924772a772e4c6e8c2c36a6b912098d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } } peers { permanent_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } } peers { permanent_uuid: "73258d93a2bd49b9b4eb548c807487c3" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } } }
I20260926 02:10:15.252686 32756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [sys.catalog]: This master's current role is: LEADER
I20260926 02:10:15.255327 302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7924772a772e4c6e8c2c36a6b912098d. Latest consensus state: current_term: 1 leader_uuid: "7924772a772e4c6e8c2c36a6b912098d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7924772a772e4c6e8c2c36a6b912098d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } } peers { permanent_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } } peers { permanent_uuid: "73258d93a2bd49b9b4eb548c807487c3" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } } }
I20260926 02:10:15.255800 302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:10:15.258023 315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7924772a772e4c6e8c2c36a6b912098d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7924772a772e4c6e8c2c36a6b912098d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } } peers { permanent_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } } peers { permanent_uuid: "73258d93a2bd49b9b4eb548c807487c3" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } } }
I20260926 02:10:15.258306 315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d [sys.catalog]: This master's current role is: LEADER
I20260926 02:10:15.262173 319 mvcc.cc:205] Tried to move back new op lower bound from 7333431767819210752 to 7333431767593508864. Current Snapshot: MvccSnapshot[applied={T|T < 7333431767819210752}]
I20260926 02:10:15.270406 302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7924772a772e4c6e8c2c36a6b912098d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7924772a772e4c6e8c2c36a6b912098d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } } peers { permanent_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } } peers { permanent_uuid: "73258d93a2bd49b9b4eb548c807487c3" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } } }
I20260926 02:10:15.270725 302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:10:15.274693 317 catalog_manager.cc:1347] Generated new cluster ID: 84fab98ed8244214a38dfb9d1d3cf31d
I20260926 02:10:15.274859 317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:10:15.276229 32742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7924772a772e4c6e8c2c36a6b912098d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7924772a772e4c6e8c2c36a6b912098d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36139 } } peers { permanent_uuid: "078c6c5cd88c4a7ca1edeb66f668f88b" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 45957 } } peers { permanent_uuid: "73258d93a2bd49b9b4eb548c807487c3" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 42219 } } }
I20260926 02:10:15.276564 32742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:10:15.312378 317 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:10:15.316370 317 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:10:15.366247 317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7924772a772e4c6e8c2c36a6b912098d: Generated new TSK 0
I20260926 02:10:15.367496 317 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:10:15.932746 32754 catalog_manager.cc:1259] Loaded cluster ID: 84fab98ed8244214a38dfb9d1d3cf31d
I20260926 02:10:15.932854 32754 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3: loading cluster ID for follower catalog manager: success
I20260926 02:10:15.935281 32754 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3: acquiring CA information for follower catalog manager: success
I20260926 02:10:15.937292 32754 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 73258d93a2bd49b9b4eb548c807487c3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:10:16.077617 313 catalog_manager.cc:1259] Loaded cluster ID: 84fab98ed8244214a38dfb9d1d3cf31d
I20260926 02:10:16.077771 313 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b: loading cluster ID for follower catalog manager: success
I20260926 02:10:16.080926 313 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b: acquiring CA information for follower catalog manager: success
I20260926 02:10:16.083076 313 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 078c6c5cd88c4a7ca1edeb66f668f88b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:10:16.530643 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 32511
I20260926 02:10:16.531913 32573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:16.643301 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 32511
I20260926 02:10:16.664603 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 32585
I20260926 02:10:16.665831 32647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:16.775480 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 32585
I20260926 02:10:16.800347 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 32660
I20260926 02:10:16.801528 32722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:16.908519 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 32660
2026-09-26T02:10:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5012 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:10:17 dist-test-slave-03rn krb5kdc[336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:10:17 dist-test-slave-03rn krb5kdc[336](info): set up 2 sockets
Sep 26 02:10:17 dist-test-slave-03rn krb5kdc[336](info): commencing operation
krb5kdc: starting...
W20260926 02:10:18.017122 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:10:18 dist-test-slave-03rn krb5kdc[336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:10:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:10:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:10:18.272819 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:35863
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:40895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:35863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:10:18.636093 349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:18.636520 349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:18.636615 349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:18.644292 349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:10:18.644418 349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:18.644474 349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:18.644526 349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:10:18.644570 349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:10:18.655223 349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:40895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:35863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:35863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:18.656675 349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:18.658038 349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:18.672042 354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:18.672111 357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:18.672009 355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:18.673537 349 server_base.cc:1034] running on GCE node
I20260926 02:10:18.674659 349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:18.676406 349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:18.677659 349 hybrid_clock.cc:648] HybridClock initialized: now 1790388618677594 us; error 86 us; skew 500 ppm
I20260926 02:10:18.678110 349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:18 dist-test-slave-03rn krb5kdc[336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388618, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:18.685504 349 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:10:18.687309 349 webserver.cc:460] Webserver started at http://127.18.159.190:36809/ using document root <none> and password file <none>
I20260926 02:10:18.688107 349 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:18.688308 349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:18.688642 349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:10:18.691846 349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "3ec94fbeb9db41559574126a7a0081fc"
format_stamp: "Formatted at 2026-09-26 02:10:18 on dist-test-slave-03rn"
server_key: "923f103c3bafeb1fd148ad3af0e8a233"
server_key_iv: "fcd45c163ea2b93c7fb1931ad44ef35b"
server_key_version: "encryptionkey@0"
I20260926 02:10:18.692752 349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "3ec94fbeb9db41559574126a7a0081fc"
format_stamp: "Formatted at 2026-09-26 02:10:18 on dist-test-slave-03rn"
server_key: "923f103c3bafeb1fd148ad3af0e8a233"
server_key_iv: "fcd45c163ea2b93c7fb1931ad44ef35b"
server_key_version: "encryptionkey@0"
I20260926 02:10:18.698766 349 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:10:18.702915 364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:18.704631 349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:10:18.704867 349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "3ec94fbeb9db41559574126a7a0081fc"
format_stamp: "Formatted at 2026-09-26 02:10:18 on dist-test-slave-03rn"
server_key: "923f103c3bafeb1fd148ad3af0e8a233"
server_key_iv: "fcd45c163ea2b93c7fb1931ad44ef35b"
server_key_version: "encryptionkey@0"
I20260926 02:10:18.705103 349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:18.748909 349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:18.753419 349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:18.753746 349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:18.773339 349 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:35863
I20260926 02:10:18.773361 415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:35863 every 8 connection(s)
I20260926 02:10:18.774848 349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:10:18.777701 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 349
I20260926 02:10:18.777953 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:10:18.778793 19070 external_mini_cluster.cc:1428] Setting key b8153a161185c135fb628710dac28819
I20260926 02:10:18.779539 416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:10:18 dist-test-slave-03rn krb5kdc[336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:10:18.796648 416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc: Bootstrap starting.
I20260926 02:10:18.801357 416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc: Neither blocks nor log segments found. Creating new log.
I20260926 02:10:18.802856 416 log.cc:824] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc: Log is configured to *not* fsync() on all Append() calls
I20260926 02:10:18.806733 416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc: No bootstrap required, opened a new log
I20260926 02:10:18.807499 419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:18.780821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40508 (local address 127.18.159.190:35863)
0926 02:10:18.781637 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:18.781649 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:18.781691 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:10:18.783267 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:18.783290 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:18.783716 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:18.784068 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:18.786087 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:18.787874 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:18.790178 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:18.791279 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:18.791485 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:18.797452 (+ 5967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:18.797465 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:18.797486 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:18.797554 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:18.801984 (+ 4430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:18.803104 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:18.803115 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:18.803124 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:18.803361 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:18.803916 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:18.803921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:18.803927 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:18.804657 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:18.804953 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:18.805605 (+ 652us) server_negotiation.cc:299] Negotiation successful
0926 02:10:18.806076 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":384,"threads_started":1}
I20260926 02:10:18.811730 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:10:18.816524 416 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec94fbeb9db41559574126a7a0081fc" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35863 } }
I20260926 02:10:18.816916 416 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:10:18.817029 416 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ec94fbeb9db41559574126a7a0081fc, State: Initialized, Role: FOLLOWER
I20260926 02:10:18.817615 416 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec94fbeb9db41559574126a7a0081fc" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35863 } }
I20260926 02:10:18.817840 416 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:10:18.817983 416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:10:18.818187 416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:10:18.820807 416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec94fbeb9db41559574126a7a0081fc" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35863 } }
I20260926 02:10:18.821343 416 leader_election.cc:302] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ec94fbeb9db41559574126a7a0081fc; no voters:
I20260926 02:10:18.821955 416 leader_election.cc:288] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:10:18.822094 421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:10:18.822885 421 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc [term 1 LEADER]: Becoming Leader. State: Replica: 3ec94fbeb9db41559574126a7a0081fc, State: Running, Role: LEADER
I20260926 02:10:18.823534 421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec94fbeb9db41559574126a7a0081fc" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35863 } }
I20260926 02:10:18.824229 416 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:10:18.826741 422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ec94fbeb9db41559574126a7a0081fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec94fbeb9db41559574126a7a0081fc" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35863 } } }
I20260926 02:10:18.826714 423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ec94fbeb9db41559574126a7a0081fc. Latest consensus state: current_term: 1 leader_uuid: "3ec94fbeb9db41559574126a7a0081fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec94fbeb9db41559574126a7a0081fc" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35863 } } }
I20260926 02:10:18.827453 423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc [sys.catalog]: This master's current role is: LEADER
I20260926 02:10:18.827447 422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc [sys.catalog]: This master's current role is: LEADER
I20260926 02:10:18.833465 428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:10:18.838239 428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:10:18.847240 428 catalog_manager.cc:1347] Generated new cluster ID: abf7ffac979645f98dac7849f808d03b
I20260926 02:10:18.847383 428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:10:18.886502 428 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:10:18.887998 428 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:10:18.894510 428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ec94fbeb9db41559574126a7a0081fc: Generated new TSK 0
I20260926 02:10:18.895431 428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:10:18.923952 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 349
I20260926 02:10:18.925220 411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:19.022584 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 349
2026-09-26T02:10:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2109 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:10:19 dist-test-slave-03rn krb5kdc[444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:10:19 dist-test-slave-03rn krb5kdc[444](info): set up 2 sockets
Sep 26 02:10:19 dist-test-slave-03rn krb5kdc[444](info): commencing operation
krb5kdc: starting...
W20260926 02:10:20.149688 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:10:20 dist-test-slave-03rn krb5kdc[444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:10:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:10:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:10:20.405000 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:32955
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:32955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:10:20.756300 457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:20.756667 457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:20.756857 457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:20.764729 457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:10:20.764856 457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:20.764933 457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:20.765029 457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:10:20.765133 457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:10:20.776744 457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:32955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:32955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:20.778264 457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:20.779731 457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:20.792781 465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:20.793005 462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:20.792889 463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:20.793756 457 server_base.cc:1034] running on GCE node
I20260926 02:10:20.794649 457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:20.796391 457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:20.797720 457 hybrid_clock.cc:648] HybridClock initialized: now 1790388620797610 us; error 94 us; skew 500 ppm
I20260926 02:10:20.798130 457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:20 dist-test-slave-03rn krb5kdc[444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388620, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:20.801738 457 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:10:20.803503 457 webserver.cc:460] Webserver started at http://127.18.159.190:44401/ using document root <none> and password file <none>
I20260926 02:10:20.804359 457 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:20.804525 457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:20.804858 457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:10:20.808290 457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "d0dd73e83ab7481c8743fa65104f9977"
format_stamp: "Formatted at 2026-09-26 02:10:20 on dist-test-slave-03rn"
server_key: "20c63c5e465302ca32b318489d64b1b2"
server_key_iv: "44c01e3601383ec721e5f0aa7dde38e7"
server_key_version: "encryptionkey@0"
I20260926 02:10:20.809238 457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "d0dd73e83ab7481c8743fa65104f9977"
format_stamp: "Formatted at 2026-09-26 02:10:20 on dist-test-slave-03rn"
server_key: "20c63c5e465302ca32b318489d64b1b2"
server_key_iv: "44c01e3601383ec721e5f0aa7dde38e7"
server_key_version: "encryptionkey@0"
I20260926 02:10:20.815567 457 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 02:10:20.820215 472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:20.822069 457 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:10:20.822278 457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "d0dd73e83ab7481c8743fa65104f9977"
format_stamp: "Formatted at 2026-09-26 02:10:20 on dist-test-slave-03rn"
server_key: "20c63c5e465302ca32b318489d64b1b2"
server_key_iv: "44c01e3601383ec721e5f0aa7dde38e7"
server_key_version: "encryptionkey@0"
I20260926 02:10:20.822495 457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:20.837893 457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:20.842700 457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:20.842975 457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:20.863868 457 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:32955
I20260926 02:10:20.863891 523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:32955 every 8 connection(s)
I20260926 02:10:20.865513 457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:10:20.869446 524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:10:20.871389 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 457
I20260926 02:10:20.871628 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:10:20.872490 19070 external_mini_cluster.cc:1428] Setting key 0aec16746c7928e018993262b74e9b98
I20260926 02:10:20.883419 524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977: Bootstrap starting.
I20260926 02:10:20.887687 524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977: Neither blocks nor log segments found. Creating new log.
Sep 26 02:10:20 dist-test-slave-03rn krb5kdc[444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:10:20.889029 524 log.cc:824] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977: Log is configured to *not* fsync() on all Append() calls
I20260926 02:10:20.892748 524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977: No bootstrap required, opened a new log
I20260926 02:10:20.899097 527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:20.874970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52818 (local address 127.18.159.190:32955)
0926 02:10:20.875937 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:20.875950 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:20.876003 (+ 53us) server_negotiation.cc:407] Connection header received
0926 02:10:20.877859 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:20.877885 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:20.878322 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:20.878692 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:20.880714 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:20.882160 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:20.884152 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:20.884966 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:20.885157 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:20.890479 (+ 5322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:20.890493 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:20.890519 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:20.890590 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:20.894166 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:20.895196 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:20.895204 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:20.895215 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:20.895417 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:20.895930 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:20.895934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:20.895936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:20.896687 (+ 751us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:20.896947 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:20.897412 (+ 465us) server_negotiation.cc:299] Negotiation successful
0926 02:10:20.897904 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":401,"threads_started":1}
I20260926 02:10:20.902033 524 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dd73e83ab7481c8743fa65104f9977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32955 } }
I20260926 02:10:20.902456 524 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:10:20.902570 524 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dd73e83ab7481c8743fa65104f9977, State: Initialized, Role: FOLLOWER
I20260926 02:10:20.903157 524 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dd73e83ab7481c8743fa65104f9977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32955 } }
I20260926 02:10:20.903407 524 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:10:20.903553 524 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:10:20.903728 524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:10:20.906059 524 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dd73e83ab7481c8743fa65104f9977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32955 } }
I20260926 02:10:20.906705 524 leader_election.cc:302] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0dd73e83ab7481c8743fa65104f9977; no voters:
I20260926 02:10:20.907377 524 leader_election.cc:288] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:10:20.907548 529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:10:20.908430 529 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 1 LEADER]: Becoming Leader. State: Replica: d0dd73e83ab7481c8743fa65104f9977, State: Running, Role: LEADER
I20260926 02:10:20.909070 529 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dd73e83ab7481c8743fa65104f9977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32955 } }
I20260926 02:10:20.909845 524 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:10:20.912101 531 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0dd73e83ab7481c8743fa65104f9977. Latest consensus state: current_term: 1 leader_uuid: "d0dd73e83ab7481c8743fa65104f9977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dd73e83ab7481c8743fa65104f9977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32955 } } }
I20260926 02:10:20.912585 531 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [sys.catalog]: This master's current role is: LEADER
I20260926 02:10:20.913182 530 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0dd73e83ab7481c8743fa65104f9977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dd73e83ab7481c8743fa65104f9977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32955 } } }
I20260926 02:10:20.913532 530 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [sys.catalog]: This master's current role is: LEADER
I20260926 02:10:20.913800 537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:10:20.930016 537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:10:20.941224 537 catalog_manager.cc:1347] Generated new cluster ID: a7d169a93dda4a9cb5de71323e415118
I20260926 02:10:20.941380 537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:10:20.971558 537 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:10:20.973112 537 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:10:20.980010 537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977: Generated new TSK 0
I20260926 02:10:20.981074 537 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.159.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
I20260926 02:10:21.040066 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:32955
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:10:21.403852 551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:21.404264 551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:21.404433 551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:21.412078 551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:21.412289 551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:21.412470 551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:10:21.423162 551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:32955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:21.424682 551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:21.426254 551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:21.440698 556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:21.440660 559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:21.440747 557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:21.446312 551 server_base.cc:1034] running on GCE node
I20260926 02:10:21.447073 551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:21.448135 551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:21.449504 551 hybrid_clock.cc:648] HybridClock initialized: now 1790388621449423 us; error 78 us; skew 500 ppm
I20260926 02:10:21.449903 551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:21 dist-test-slave-03rn krb5kdc[444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388621, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:21.453466 551 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:10:21.455323 551 webserver.cc:460] Webserver started at http://127.18.159.129:44783/ using document root <none> and password file <none>
I20260926 02:10:21.456264 551 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:21.456418 551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:21.456745 551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:10:21.459884 551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "8911d79cd5134be8a7cf8838834373d9"
format_stamp: "Formatted at 2026-09-26 02:10:21 on dist-test-slave-03rn"
server_key: "c12269d9153cba41c1d7521fc5a63e15"
server_key_iv: "157d4f2bfe326e5342344ce7e03999fb"
server_key_version: "encryptionkey@0"
I20260926 02:10:21.460814 551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "8911d79cd5134be8a7cf8838834373d9"
format_stamp: "Formatted at 2026-09-26 02:10:21 on dist-test-slave-03rn"
server_key: "c12269d9153cba41c1d7521fc5a63e15"
server_key_iv: "157d4f2bfe326e5342344ce7e03999fb"
server_key_version: "encryptionkey@0"
I20260926 02:10:21.466569 551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:10:21.470613 566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:21.472273 551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:10:21.472478 551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "8911d79cd5134be8a7cf8838834373d9"
format_stamp: "Formatted at 2026-09-26 02:10:21 on dist-test-slave-03rn"
server_key: "c12269d9153cba41c1d7521fc5a63e15"
server_key_iv: "157d4f2bfe326e5342344ce7e03999fb"
server_key_version: "encryptionkey@0"
I20260926 02:10:21.472730 551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:21.530093 551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:21.535772 551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:21.536147 551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:21.537400 551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:10:21.538976 551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:10:21.539053 551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:21.539198 551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:10:21.539260 551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:21.578575 551 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:40959
I20260926 02:10:21.578639 678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:40959 every 8 connection(s)
I20260926 02:10:21.580053 551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:10:21.588603 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 551
I20260926 02:10:21.588836 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:10:21.589636 19070 external_mini_cluster.cc:1428] Setting key eb0843f33f16906bebfd7835ef8c143f
Sep 26 02:10:21 dist-test-slave-03rn krb5kdc[444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388621, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:10:21.602584 681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:21.582868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:47065 (local address 127.18.159.190:32955)
0926 02:10:21.583375 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:21.583383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:21.584498 (+ 1115us) server_negotiation.cc:407] Connection header received
0926 02:10:21.586325 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:21.586329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:21.586403 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:21.586504 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:21.588097 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:21.589050 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:21.591291 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:21.591822 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:21.591972 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:21.596254 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:21.596261 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:21.596264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:21.596306 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:21.599128 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:21.600035 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:21.600040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:21.600046 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:21.600237 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:21.600712 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:21.600718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:21.600721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:21.601004 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:21.601159 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:21.601942 (+ 783us) server_negotiation.cc:299] Negotiation successful
0926 02:10:21.602199 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":237,"threads_started":1}
I20260926 02:10:21.603653 682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:21.583463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.129:47065)
0926 02:10:21.584323 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:21.584370 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:21.586052 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:21.586746 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:21.586754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:21.587293 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:21.587897 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:21.587914 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:21.589221 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:21.589228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:21.591138 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:21.591154 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:21.592077 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:21.592086 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:21.592380 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:21.593220 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:21.593245 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:21.595880 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:21.599341 (+ 3461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:21.599354 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:21.599380 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:21.599884 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:21.600359 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:21.600364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:21.600368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:21.600587 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:21.601155 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:21.601165 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:21.601536 (+ 371us) client_negotiation.cc:769] Sending connection context
0926 02:10:21.601834 (+ 298us) client_negotiation.cc:240] Negotiation successful
0926 02:10:21.602474 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":321,"threads_started":1}
I20260926 02:10:21.605482 679 heartbeater.cc:347] Connected to a master server at 127.18.159.190:32955
I20260926 02:10:21.605837 679 heartbeater.cc:464] Registering TS with master...
I20260926 02:10:21.606622 679 heartbeater.cc:511] Master 127.18.159.190:32955 requested a full tablet report, sending...
I20260926 02:10:21.609216 489 ts_manager.cc:194] Registered new tserver with Master: 8911d79cd5134be8a7cf8838834373d9 (127.18.159.129:40959)
I20260926 02:10:21.611573 489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:47065
WARNING: no policy specified for kudu/127.18.159.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
I20260926 02:10:21.645612 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:32955
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:10:21.986598 685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:21.986912 685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:21.987075 685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:21.994897 685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:21.995030 685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:21.995213 685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:10:22.005582 685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:32955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:22.006968 685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:22.008498 685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:22.022666 690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:22.022616 693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:22.023141 691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:22.023939 685 server_base.cc:1034] running on GCE node
I20260926 02:10:22.024729 685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:22.025882 685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:22.027115 685 hybrid_clock.cc:648] HybridClock initialized: now 1790388622027066 us; error 46 us; skew 500 ppm
I20260926 02:10:22.027460 685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:22 dist-test-slave-03rn krb5kdc[444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388622, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:22.030786 685 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:10:22.032543 685 webserver.cc:460] Webserver started at http://127.18.159.130:35661/ using document root <none> and password file <none>
I20260926 02:10:22.033370 685 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:22.033533 685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:22.033877 685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:10:22.037144 685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "9af40ba88bdc4287a6f472bfa69fc076"
format_stamp: "Formatted at 2026-09-26 02:10:22 on dist-test-slave-03rn"
server_key: "379a341e52e62de094db00a685fd168b"
server_key_iv: "b0cf1595c8b5d2a761791633d917429e"
server_key_version: "encryptionkey@0"
I20260926 02:10:22.038017 685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "9af40ba88bdc4287a6f472bfa69fc076"
format_stamp: "Formatted at 2026-09-26 02:10:22 on dist-test-slave-03rn"
server_key: "379a341e52e62de094db00a685fd168b"
server_key_iv: "b0cf1595c8b5d2a761791633d917429e"
server_key_version: "encryptionkey@0"
I20260926 02:10:22.043560 685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 02:10:22.047356 700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:22.048977 685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:10:22.049185 685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "9af40ba88bdc4287a6f472bfa69fc076"
format_stamp: "Formatted at 2026-09-26 02:10:22 on dist-test-slave-03rn"
server_key: "379a341e52e62de094db00a685fd168b"
server_key_iv: "b0cf1595c8b5d2a761791633d917429e"
server_key_version: "encryptionkey@0"
I20260926 02:10:22.049397 685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:22.070734 685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:22.075368 685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:22.075745 685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:22.076880 685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:10:22.078440 685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:10:22.078553 685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:22.078711 685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:10:22.078780 685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:22.114924 685 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:38413
I20260926 02:10:22.114959 812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:38413 every 8 connection(s)
I20260926 02:10:22.116317 685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:10:22.120074 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 685
I20260926 02:10:22.120281 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:10:22.120819 19070 external_mini_cluster.cc:1428] Setting key 1db01e3478cc07cabef12a8cafd73ca1
Sep 26 02:10:22 dist-test-slave-03rn krb5kdc[444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388622, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:10:22.138340 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:22.118921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:48047 (local address 127.18.159.190:32955)
0926 02:10:22.119378 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:22.119392 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:22.120566 (+ 1174us) server_negotiation.cc:407] Connection header received
0926 02:10:22.122696 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:22.122701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:22.122928 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:22.123070 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:22.124736 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.125521 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.127440 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.127900 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.128052 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:22.131999 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:22.132005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:22.132008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:22.132045 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:22.134890 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:22.135767 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:22.135771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:22.135774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:22.135905 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:22.136459 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:22.136463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:22.136465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:22.136757 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:22.136917 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:22.137690 (+ 773us) server_negotiation.cc:299] Negotiation successful
0926 02:10:22.137937 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":228,"threads_started":1}
I20260926 02:10:22.139484 816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:22.119494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.130:48047)
0926 02:10:22.120396 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:22.120443 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:22.122328 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:22.123327 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:22.123336 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:22.123869 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:22.124559 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:22.124577 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.125675 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.125679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:22.127280 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:22.127293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.128138 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.128146 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:22.128436 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:22.129152 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:22.129179 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:22.131721 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:22.135068 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:22.135078 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:22.135099 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:22.135610 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:22.136025 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:22.136030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:22.136034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:22.136328 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:22.136892 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:22.136900 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:22.137296 (+ 396us) client_negotiation.cc:769] Sending connection context
0926 02:10:22.137587 (+ 291us) client_negotiation.cc:240] Negotiation successful
0926 02:10:22.138176 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":323,"threads_started":1}
I20260926 02:10:22.141196 813 heartbeater.cc:347] Connected to a master server at 127.18.159.190:32955
I20260926 02:10:22.141584 813 heartbeater.cc:464] Registering TS with master...
I20260926 02:10:22.142590 813 heartbeater.cc:511] Master 127.18.159.190:32955 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.159.131@KRBTEST.COM; defaulting to no policy
I20260926 02:10:22.144357 489 ts_manager.cc:194] Registered new tserver with Master: 9af40ba88bdc4287a6f472bfa69fc076 (127.18.159.130:38413)
I20260926 02:10:22.145223 489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:48047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
I20260926 02:10:22.175251 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:32955
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:10:22.517041 819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:22.517341 819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:22.517445 819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:22.524935 819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:22.525061 819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:22.525178 819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:10:22.535358 819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:32955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:22.536798 819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:22.538267 819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:22.551908 825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:22.551947 824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:22.551862 827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:22.552734 819 server_base.cc:1034] running on GCE node
I20260926 02:10:22.553476 819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:22.554569 819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:22.555774 819 hybrid_clock.cc:648] HybridClock initialized: now 1790388622555734 us; error 41 us; skew 500 ppm
I20260926 02:10:22.556070 819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:22 dist-test-slave-03rn krb5kdc[444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388622, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:22.559628 819 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:10:22.561326 819 webserver.cc:460] Webserver started at http://127.18.159.131:39045/ using document root <none> and password file <none>
I20260926 02:10:22.562096 819 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:22.562232 819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:22.562577 819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:10:22.565801 819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "2601425df57849a6bbc056dcbf1ea73d"
format_stamp: "Formatted at 2026-09-26 02:10:22 on dist-test-slave-03rn"
server_key: "49454775f30aca5cf742b98b19d53ef4"
server_key_iv: "3115ddd3aef163aba5bba9d16edeb9cb"
server_key_version: "encryptionkey@0"
I20260926 02:10:22.566646 819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "2601425df57849a6bbc056dcbf1ea73d"
format_stamp: "Formatted at 2026-09-26 02:10:22 on dist-test-slave-03rn"
server_key: "49454775f30aca5cf742b98b19d53ef4"
server_key_iv: "3115ddd3aef163aba5bba9d16edeb9cb"
server_key_version: "encryptionkey@0"
I20260926 02:10:22.572293 819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:10:22.575984 834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:22.577592 819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:10:22.577925 819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "2601425df57849a6bbc056dcbf1ea73d"
format_stamp: "Formatted at 2026-09-26 02:10:22 on dist-test-slave-03rn"
server_key: "49454775f30aca5cf742b98b19d53ef4"
server_key_iv: "3115ddd3aef163aba5bba9d16edeb9cb"
server_key_version: "encryptionkey@0"
I20260926 02:10:22.578190 819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:22.594385 819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:22.599330 819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:22.599740 819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:22.600900 819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:10:22.602468 819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:10:22.602602 819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:22.602779 819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:10:22.602865 819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:22.615217 679 heartbeater.cc:503] Master 127.18.159.190:32955 was elected leader, sending a full tablet report...
I20260926 02:10:22.640144 819 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:35439
I20260926 02:10:22.640257 946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:35439 every 8 connection(s)
I20260926 02:10:22.641652 819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:10:22.649250 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 819
I20260926 02:10:22.649444 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:10:22.650038 19070 external_mini_cluster.cc:1428] Setting key 636f6d5fd920e076dd6893a133ff14de
Sep 26 02:10:22 dist-test-slave-03rn krb5kdc[444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388622, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:10:22.663300 949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:22.644275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:59447 (local address 127.18.159.190:32955)
0926 02:10:22.644753 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:22.644760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:22.645799 (+ 1039us) server_negotiation.cc:407] Connection header received
0926 02:10:22.647708 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:22.647713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:22.647800 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:22.647903 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:22.649783 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.650856 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.652982 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.653519 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.653689 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:22.657442 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:22.657448 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:22.657451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:22.657487 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:22.660032 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:22.660953 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:22.660957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:22.660960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:22.661122 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:22.661554 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:22.661558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:22.661560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:22.661849 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:22.661971 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:22.662805 (+ 834us) server_negotiation.cc:299] Negotiation successful
0926 02:10:22.663005 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":207,"threads_started":1}
I20260926 02:10:22.664525 950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:22.644857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.131:59447)
0926 02:10:22.645631 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:22.645673 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:22.647413 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:22.648136 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:22.648145 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:22.648817 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:22.649573 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:22.649603 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.651040 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.651046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:22.652835 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:22.652849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.653782 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.653788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:22.654025 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:22.654701 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:22.654726 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:22.657188 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:22.660250 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:22.660258 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:22.660273 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:22.660800 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:22.661227 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:22.661230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:22.661232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:22.661440 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:22.661991 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:22.661997 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:22.662365 (+ 368us) client_negotiation.cc:769] Sending connection context
0926 02:10:22.662656 (+ 291us) client_negotiation.cc:240] Negotiation successful
0926 02:10:22.663086 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":286,"threads_started":1}
I20260926 02:10:22.666157 947 heartbeater.cc:347] Connected to a master server at 127.18.159.190:32955
I20260926 02:10:22.666517 947 heartbeater.cc:464] Registering TS with master...
I20260926 02:10:22.667311 947 heartbeater.cc:511] Master 127.18.159.190:32955 requested a full tablet report, sending...
I20260926 02:10:22.668975 488 ts_manager.cc:194] Registered new tserver with Master: 2601425df57849a6bbc056dcbf1ea73d (127.18.159.131:35439)
I20260926 02:10:22.669855 488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:59447
I20260926 02:10:22.679296 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:10:22.697834 949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:22.684085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52822 (local address 127.18.159.190:32955)
0926 02:10:22.684314 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:22.684322 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:22.684676 (+ 354us) server_negotiation.cc:407] Connection header received
0926 02:10:22.684887 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:22.684891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:22.684972 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:22.685090 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:22.686506 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.687273 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.689113 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.689540 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.689689 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:22.692306 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:22.692311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:22.692315 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:22.692355 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:22.694911 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:22.695682 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:22.695692 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:22.695695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:22.695825 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:22.696275 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:22.696286 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:22.696289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:22.696636 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:22.696916 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:22.697246 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 02:10:22.697503 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 02:10:22.702560 488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:10:22.710551 488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:10:22.744774 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:22.724694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60812 (local address 127.18.159.129:40959)
0926 02:10:22.735078 (+ 10384us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:22.735092 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:22.735136 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:10:22.735293 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:22.735306 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:22.735696 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:22.735956 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:10:22.737074 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.738550 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.741618 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.743388 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.743647 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:22.743854 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:22.744142 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 02:10:22.744378 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10131,"thread_start_us":9907,"threads_started":1}
I20260926 02:10:22.744828 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:22.724401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40959 (local address 127.0.0.1:60812)
0926 02:10:22.725175 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:22.725219 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:22.726547 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:22.736210 (+ 9663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:22.736221 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:22.736272 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:10:22.736808 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:22.736828 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.738760 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.738765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:22.741394 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:22.741414 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.743631 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.743638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:22.743840 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:22.743894 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 02:10:22.744029 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 02:10:22.744292 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":5,"thread_start_us":331,"threads_started":1}
I20260926 02:10:22.745760 963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:22.726718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:38413 (local address 127.0.0.1:34908)
0926 02:10:22.736184 (+ 9466us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:22.736215 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:22.736499 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:22.737582 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:22.737587 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:22.737613 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:10:22.738087 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:22.738104 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.740053 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.740058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:22.742295 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:22.742316 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.744710 (+ 2394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.744718 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:22.744961 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:22.745006 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 02:10:22.745131 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 02:10:22.745316 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9238,"mutex_wait_us":89,"thread_start_us":9619,"threads_started":1}
I20260926 02:10:22.746323 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:22.727140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.18.159.130:38413)
0926 02:10:22.734553 (+ 7413us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:22.734578 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:22.736360 (+ 1782us) server_negotiation.cc:407] Connection header received
0926 02:10:22.736754 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:22.736763 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:22.737051 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:22.737241 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:10:22.738286 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.739877 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.742516 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.744146 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.744347 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:22.744537 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:22.745596 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0926 02:10:22.745812 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7119,"thread_start_us":6941,"threads_started":1}
I20260926 02:10:22.746917 965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:22.727325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36232 (local address 127.18.159.131:35439)
0926 02:10:22.735472 (+ 8147us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:22.735485 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:22.737019 (+ 1534us) server_negotiation.cc:407] Connection header received
0926 02:10:22.737361 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:22.737369 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:22.737629 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:22.737826 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:10:22.738737 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.739909 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.742957 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.744315 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.744501 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:22.744671 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:22.746396 (+ 1725us) server_negotiation.cc:299] Negotiation successful
0926 02:10:22.746588 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7926,"thread_start_us":7745,"threads_started":1}
I20260926 02:10:22.747340 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:22.726918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:35439 (local address 127.0.0.1:36232)
0926 02:10:22.736898 (+ 9980us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:22.736917 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:22.737143 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:22.738197 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:22.738205 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:22.738235 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:10:22.738545 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:22.738553 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.740102 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.740107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:22.742763 (+ 2656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:22.742777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:22.744842 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:22.744852 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:22.746174 (+ 1322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:22.746216 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 02:10:22.746327 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 02:10:22.746676 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9805,"mutex_wait_us":2,"thread_start_us":8803,"threads_started":1}
I20260926 02:10:22.749639 614 tablet_service.cc:1467] Processing CreateTablet for tablet cab6d805d6244d808573411afd3674b9 (DEFAULT_TABLE table=test-table [id=dfd65ce2d41046e6ba08cd4c1a257191]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:10:22.750058 748 tablet_service.cc:1467] Processing CreateTablet for tablet cab6d805d6244d808573411afd3674b9 (DEFAULT_TABLE table=test-table [id=dfd65ce2d41046e6ba08cd4c1a257191]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:10:22.751268 614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cab6d805d6244d808573411afd3674b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:10:22.751370 748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cab6d805d6244d808573411afd3674b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:10:22.751170 882 tablet_service.cc:1467] Processing CreateTablet for tablet cab6d805d6244d808573411afd3674b9 (DEFAULT_TABLE table=test-table [id=dfd65ce2d41046e6ba08cd4c1a257191]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:10:22.752527 882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cab6d805d6244d808573411afd3674b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:10:22.766970 966 tablet_bootstrap.cc:492] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d: Bootstrap starting.
I20260926 02:10:22.767688 968 tablet_bootstrap.cc:492] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076: Bootstrap starting.
I20260926 02:10:22.771605 968 tablet_bootstrap.cc:654] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076: Neither blocks nor log segments found. Creating new log.
I20260926 02:10:22.771613 966 tablet_bootstrap.cc:654] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d: Neither blocks nor log segments found. Creating new log.
I20260926 02:10:22.772691 967 tablet_bootstrap.cc:492] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9: Bootstrap starting.
I20260926 02:10:22.773176 968 log.cc:824] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076: Log is configured to *not* fsync() on all Append() calls
I20260926 02:10:22.773176 966 log.cc:824] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:10:22.777024 968 tablet_bootstrap.cc:492] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076: No bootstrap required, opened a new log
I20260926 02:10:22.777040 966 tablet_bootstrap.cc:492] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d: No bootstrap required, opened a new log
I20260926 02:10:22.777051 967 tablet_bootstrap.cc:654] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9: Neither blocks nor log segments found. Creating new log.
I20260926 02:10:22.777341 966 ts_tablet_manager.cc:1397] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260926 02:10:22.777354 968 ts_tablet_manager.cc:1397] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 02:10:22.778689 967 log.cc:824] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9: Log is configured to *not* fsync() on all Append() calls
I20260926 02:10:22.782526 967 tablet_bootstrap.cc:492] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9: No bootstrap required, opened a new log
I20260926 02:10:22.782855 967 ts_tablet_manager.cc:1397] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 02:10:22.786645 968 raft_consensus.cc:348] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8911d79cd5134be8a7cf8838834373d9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40959 } } peers { permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 } } peers { permanent_uuid: "2601425df57849a6bbc056dcbf1ea73d" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35439 } }
I20260926 02:10:22.787091 968 raft_consensus.cc:374] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:10:22.787194 968 raft_consensus.cc:729] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9af40ba88bdc4287a6f472bfa69fc076, State: Initialized, Role: FOLLOWER
I20260926 02:10:22.787911 968 consensus_queue.cc:260] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8911d79cd5134be8a7cf8838834373d9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40959 } } peers { permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 } } peers { permanent_uuid: "2601425df57849a6bbc056dcbf1ea73d" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35439 } }
I20260926 02:10:22.788333 966 raft_consensus.cc:348] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8911d79cd5134be8a7cf8838834373d9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40959 } } peers { permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 } } peers { permanent_uuid: "2601425df57849a6bbc056dcbf1ea73d" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35439 } }
I20260926 02:10:22.788816 966 raft_consensus.cc:374] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:10:22.788914 966 raft_consensus.cc:729] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2601425df57849a6bbc056dcbf1ea73d, State: Initialized, Role: FOLLOWER
I20260926 02:10:22.789348 813 heartbeater.cc:503] Master 127.18.159.190:32955 was elected leader, sending a full tablet report...
I20260926 02:10:22.789647 966 consensus_queue.cc:260] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8911d79cd5134be8a7cf8838834373d9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40959 } } peers { permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 } } peers { permanent_uuid: "2601425df57849a6bbc056dcbf1ea73d" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35439 } }
I20260926 02:10:22.791982 968 ts_tablet_manager.cc:1428] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260926 02:10:22.793040 947 heartbeater.cc:503] Master 127.18.159.190:32955 was elected leader, sending a full tablet report...
I20260926 02:10:22.793679 966 ts_tablet_manager.cc:1428] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260926 02:10:22.796023 967 raft_consensus.cc:348] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8911d79cd5134be8a7cf8838834373d9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40959 } } peers { permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 } } peers { permanent_uuid: "2601425df57849a6bbc056dcbf1ea73d" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35439 } }
I20260926 02:10:22.796604 967 raft_consensus.cc:374] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:10:22.796764 967 raft_consensus.cc:729] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8911d79cd5134be8a7cf8838834373d9, State: Initialized, Role: FOLLOWER
I20260926 02:10:22.797379 967 consensus_queue.cc:260] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8911d79cd5134be8a7cf8838834373d9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40959 } } peers { permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 } } peers { permanent_uuid: "2601425df57849a6bbc056dcbf1ea73d" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35439 } }
I20260926 02:10:22.799263 967 ts_tablet_manager.cc:1428] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9: Time spent starting tablet: real 0.016s user 0.008s sys 0.008s
W20260926 02:10:22.832656 680 tablet.cc:2369] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:10:22.868446 814 tablet.cc:2369] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:10:22.893440 948 tablet.cc:2369] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:10:23.068485 973 raft_consensus.cc:482] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:10:23.068935 973 raft_consensus.cc:504] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8911d79cd5134be8a7cf8838834373d9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40959 } } peers { permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 } } peers { permanent_uuid: "2601425df57849a6bbc056dcbf1ea73d" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35439 } }
I20260926 02:10:23.070755 973 leader_election.cc:288] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8911d79cd5134be8a7cf8838834373d9 (127.18.159.129:40959), 9af40ba88bdc4287a6f472bfa69fc076 (127.18.159.130:38413)
I20260926 02:10:23.077502 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:23.071187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:55489 (local address 127.18.159.129:40959)
0926 02:10:23.071401 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:23.071407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:23.071430 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:10:23.071672 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:23.071680 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:23.071785 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:23.071900 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:10:23.072508 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:23.073516 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:23.075501 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:23.076618 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:23.076795 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:23.076853 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:23.077156 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 02:10:23.077249 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:10:23.077584 950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:23.071065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40959 (local address 127.18.159.131:55489)
0926 02:10:23.071305 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:23.071328 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:23.071471 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:23.072090 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:23.072093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:23.072116 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:10:23.072333 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:23.072341 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:23.073663 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:23.073668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:23.075299 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:23.075329 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:23.076781 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:23.076788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:23.076987 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:23.077024 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 02:10:23.077128 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 02:10:23.077270 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":1}
I20260926 02:10:23.078521 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:23.071831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:56721 (local address 127.18.159.130:38413)
0926 02:10:23.072057 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:23.072063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:23.072347 (+ 284us) server_negotiation.cc:407] Connection header received
0926 02:10:23.072526 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:23.072530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:23.072590 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:23.072656 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:10:23.073240 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:23.074227 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:23.076284 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:23.077541 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:23.077700 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:23.077769 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:23.078225 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 02:10:23.078342 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:10:23.078632 975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:23.071533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:38413 (local address 127.18.159.131:56721)
0926 02:10:23.072237 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:23.072254 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:23.072399 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:23.072702 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:23.072706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:23.072731 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:10:23.073071 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:23.073082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:23.074391 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:23.074395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:23.076070 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:23.076086 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:23.077899 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:23.077905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:23.078074 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:23.078097 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:10:23.078170 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 02:10:23.078294 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":312,"threads_started":1}
I20260926 02:10:23.078806 634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6d805d6244d808573411afd3674b9" candidate_uuid: "2601425df57849a6bbc056dcbf1ea73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8911d79cd5134be8a7cf8838834373d9" is_pre_election: true
I20260926 02:10:23.079491 634 raft_consensus.cc:2454] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2601425df57849a6bbc056dcbf1ea73d in term 0.
I20260926 02:10:23.079761 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6d805d6244d808573411afd3674b9" candidate_uuid: "2601425df57849a6bbc056dcbf1ea73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9af40ba88bdc4287a6f472bfa69fc076" is_pre_election: true
I20260926 02:10:23.080302 836 leader_election.cc:302] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2601425df57849a6bbc056dcbf1ea73d, 8911d79cd5134be8a7cf8838834373d9; no voters:
I20260926 02:10:23.080353 768 raft_consensus.cc:2454] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2601425df57849a6bbc056dcbf1ea73d in term 0.
I20260926 02:10:23.080770 973 raft_consensus.cc:2781] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:10:23.080966 973 raft_consensus.cc:482] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:10:23.081086 973 raft_consensus.cc:3037] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:10:23.083470 973 raft_consensus.cc:504] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8911d79cd5134be8a7cf8838834373d9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40959 } } peers { permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 } } peers { permanent_uuid: "2601425df57849a6bbc056dcbf1ea73d" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35439 } }
I20260926 02:10:23.084196 973 leader_election.cc:288] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d [CANDIDATE]: Term 1 election: Requested vote from peers 8911d79cd5134be8a7cf8838834373d9 (127.18.159.129:40959), 9af40ba88bdc4287a6f472bfa69fc076 (127.18.159.130:38413)
I20260926 02:10:23.084667 634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6d805d6244d808573411afd3674b9" candidate_uuid: "2601425df57849a6bbc056dcbf1ea73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8911d79cd5134be8a7cf8838834373d9"
I20260926 02:10:23.084781 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab6d805d6244d808573411afd3674b9" candidate_uuid: "2601425df57849a6bbc056dcbf1ea73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9af40ba88bdc4287a6f472bfa69fc076"
I20260926 02:10:23.084892 634 raft_consensus.cc:3037] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:10:23.085029 768 raft_consensus.cc:3037] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:10:23.087176 634 raft_consensus.cc:2454] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2601425df57849a6bbc056dcbf1ea73d in term 1.
I20260926 02:10:23.087246 768 raft_consensus.cc:2454] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2601425df57849a6bbc056dcbf1ea73d in term 1.
I20260926 02:10:23.087610 836 leader_election.cc:302] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2601425df57849a6bbc056dcbf1ea73d, 8911d79cd5134be8a7cf8838834373d9; no voters:
I20260926 02:10:23.087972 973 raft_consensus.cc:2781] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:10:23.088418 973 raft_consensus.cc:686] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d [term 1 LEADER]: Becoming Leader. State: Replica: 2601425df57849a6bbc056dcbf1ea73d, State: Running, Role: LEADER
I20260926 02:10:23.089021 973 consensus_queue.cc:237] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8911d79cd5134be8a7cf8838834373d9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40959 } } peers { permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 } } peers { permanent_uuid: "2601425df57849a6bbc056dcbf1ea73d" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35439 } }
I20260926 02:10:23.093684 488 catalog_manager.cc:5495] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2601425df57849a6bbc056dcbf1ea73d (127.18.159.131). New cstate: current_term: 1 leader_uuid: "2601425df57849a6bbc056dcbf1ea73d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8911d79cd5134be8a7cf8838834373d9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2601425df57849a6bbc056dcbf1ea73d" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35439 } health_report { overall_health: HEALTHY } } }
I20260926 02:10:23.195915 965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:23.189963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36238 (local address 127.18.159.131:35439)
0926 02:10:23.190160 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:23.190165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:23.190184 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:10:23.190375 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:23.190379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:23.190467 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:23.190601 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:10:23.190963 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:23.191761 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:23.193448 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:23.194006 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:23.194120 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:23.194710 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:10:23.195261 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:10:23.195397 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:23.195516 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 02:10:23.195593 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:10:23.204581 634 raft_consensus.cc:1261] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9 [term 1 FOLLOWER]: Refusing update from remote peer 2601425df57849a6bbc056dcbf1ea73d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:10:23.204650 768 raft_consensus.cc:1261] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076 [term 1 FOLLOWER]: Refusing update from remote peer 2601425df57849a6bbc056dcbf1ea73d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:10:23.205626 976 consensus_queue.cc:1035] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8911d79cd5134be8a7cf8838834373d9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:10:23.206017 973 consensus_queue.cc:1035] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:10:23.218727 979 mvcc.cc:205] Tried to move back new op lower bound from 7333431800634552320 to 7333431800176939008. Current Snapshot: MvccSnapshot[applied={T|T < 7333431800634552320}]
I20260926 02:10:23.236528 489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:10:23.236883 489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:10:23.240936 489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977: Sending DeleteTablet for 3 replicas of tablet cab6d805d6244d808573411afd3674b9
I20260926 02:10:23.242265 882 tablet_service.cc:1514] Processing DeleteTablet for tablet cab6d805d6244d808573411afd3674b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:10:23 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:36232
I20260926 02:10:23.242403 748 tablet_service.cc:1514] Processing DeleteTablet for tablet cab6d805d6244d808573411afd3674b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:10:23 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:34908
I20260926 02:10:23.242388 614 tablet_service.cc:1514] Processing DeleteTablet for tablet cab6d805d6244d808573411afd3674b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:10:23 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:60812
I20260926 02:10:23.243249 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 551
I20260926 02:10:23.255863 991 tablet_replica.cc:330] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9: stopping tablet replica
I20260926 02:10:23.257174 991 raft_consensus.cc:2229] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:10:23.257362 990 tablet_replica.cc:330] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076: stopping tablet replica
I20260926 02:10:23.257840 991 raft_consensus.cc:2258] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:10:23.257856 989 tablet_replica.cc:330] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d: stopping tablet replica
I20260926 02:10:23.257846 990 raft_consensus.cc:2229] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:10:23.258306 989 raft_consensus.cc:2229] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:10:23.258345 990 raft_consensus.cc:2258] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:10:23.258992 989 raft_consensus.cc:2258] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:10:23.260725 991 ts_tablet_manager.cc:1905] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:10:23.260708 990 ts_tablet_manager.cc:1905] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:10:23.268267 990 ts_tablet_manager.cc:1918] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:10:23.268258 991 ts_tablet_manager.cc:1918] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:10:23.268477 991 log.cc:1196] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/cab6d805d6244d808573411afd3674b9
I20260926 02:10:23.268477 990 log.cc:1196] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/cab6d805d6244d808573411afd3674b9
I20260926 02:10:23.269100 990 ts_tablet_manager.cc:1939] T cab6d805d6244d808573411afd3674b9 P 9af40ba88bdc4287a6f472bfa69fc076: Deleting consensus metadata
I20260926 02:10:23.269100 991 ts_tablet_manager.cc:1939] T cab6d805d6244d808573411afd3674b9 P 8911d79cd5134be8a7cf8838834373d9: Deleting consensus metadata
Sep 26 02:10:23 dist-test-slave-03rn krb5kdc[444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:10:23.271139 474 catalog_manager.cc:4841] TS 9af40ba88bdc4287a6f472bfa69fc076 (127.18.159.130:38413): tablet cab6d805d6244d808573411afd3674b9 (table test-table [id=dfd65ce2d41046e6ba08cd4c1a257191]) successfully deleted
I20260926 02:10:23.271577 474 catalog_manager.cc:4841] TS 8911d79cd5134be8a7cf8838834373d9 (127.18.159.129:40959): tablet cab6d805d6244d808573411afd3674b9 (table test-table [id=dfd65ce2d41046e6ba08cd4c1a257191]) successfully deleted
I20260926 02:10:23.273464 989 ts_tablet_manager.cc:1905] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:10:23.279837 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:23.244368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60824 (local address 127.18.159.129:40959)
0926 02:10:23.244646 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:23.244656 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:23.257152 (+ 12496us) server_negotiation.cc:407] Connection header received
0926 02:10:23.257694 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:23.257702 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:23.257808 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:23.257899 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:23.260420 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:23.261575 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:23.263813 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:23.264574 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:23.264693 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:23.271638 (+ 6945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:23.271658 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:23.271678 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:23.271730 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:23.276071 (+ 4341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:23.277226 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:23.277238 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:23.277241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:23.277440 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:23.277942 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:23.277948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:23.277951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:23.278432 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:23.278791 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:23.279156 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 02:10:23.279420 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 02:10:23.280541 989 ts_tablet_manager.cc:1918] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:10:23.280687 674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:23.280761 989 log.cc:1196] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/cab6d805d6244d808573411afd3674b9
I20260926 02:10:23.281657 989 ts_tablet_manager.cc:1939] T cab6d805d6244d808573411afd3674b9 P 2601425df57849a6bbc056dcbf1ea73d: Deleting consensus metadata
I20260926 02:10:23.284288 473 catalog_manager.cc:4841] TS 2601425df57849a6bbc056dcbf1ea73d (127.18.159.131:35439): tablet cab6d805d6244d808573411afd3674b9 (table test-table [id=dfd65ce2d41046e6ba08cd4c1a257191]) successfully deleted
I20260926 02:10:23.401237 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 551
I20260926 02:10:23.424964 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 685
Sep 26 02:10:23 dist-test-slave-03rn krb5kdc[444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.130@KRBTEST.COM
I20260926 02:10:23.443104 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:23.425922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34910 (local address 127.18.159.130:38413)
0926 02:10:23.426118 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:23.426124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:23.426183 (+ 59us) server_negotiation.cc:407] Connection header received
0926 02:10:23.426435 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:23.426441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:23.426533 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:23.426649 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:23.428214 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:23.429331 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:23.430605 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:23.431257 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:23.431390 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:23.436022 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:23.436045 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:23.436067 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:23.436118 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:23.439905 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:23.440811 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:23.440821 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:23.440825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:23.440985 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:23.441351 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:23.441356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:23.441359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:23.441856 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:23.442146 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:23.442488 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 02:10:23.442735 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:10:23.443861 808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:23.534912 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 685
I20260926 02:10:23.552341 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 819
Sep 26 02:10:23 dist-test-slave-03rn krb5kdc[444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.131@KRBTEST.COM
I20260926 02:10:23.571094 965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:23.553323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36240 (local address 127.18.159.131:35439)
0926 02:10:23.553495 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:23.553499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:23.553518 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:10:23.553710 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:23.553714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:23.553784 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:23.553866 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:23.555656 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:23.556633 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:23.558527 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:23.558986 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:23.559089 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:23.564703 (+ 5614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:23.564718 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:23.564740 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:23.564788 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:23.567965 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:23.568826 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:23.568834 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:23.568836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:23.569014 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:23.569408 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:23.569417 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:23.569424 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:23.569772 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:23.570106 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:23.570426 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 02:10:23.570672 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:10:23.571820 942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:23.694582 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 819
I20260926 02:10:23.716346 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 457
I20260926 02:10:23.717499 519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:23.829567 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 457
I20260926 02:10:23.851135 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:32955
--webserver_interface=127.18.159.190
--webserver_port=44401
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:32955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:10:24.198886 997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:24.199265 997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:24.199460 997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:24.207101 997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:10:24.207286 997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:24.207406 997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:24.207518 997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:10:24.207604 997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:10:24.218014 997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:32955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:32955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=44401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:24.219462 997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:24.220903 997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:24.245244 1006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:24.245239 1004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:24.245306 1003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:24.246402 997 server_base.cc:1034] running on GCE node
I20260926 02:10:24.247398 997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:24.249264 997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:24.250566 997 hybrid_clock.cc:648] HybridClock initialized: now 1790388624250497 us; error 59 us; skew 500 ppm
I20260926 02:10:24.251008 997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:24 dist-test-slave-03rn krb5kdc[444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388624, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:24.259106 997 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:10:24.261303 997 webserver.cc:460] Webserver started at http://127.18.159.190:44401/ using document root <none> and password file <none>
I20260926 02:10:24.262207 997 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:24.262395 997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:24.269635 997 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:10:24.273227 1013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:24.274897 997 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:10:24.275089 997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "d0dd73e83ab7481c8743fa65104f9977"
format_stamp: "Formatted at 2026-09-26 02:10:20 on dist-test-slave-03rn"
server_key: "20c63c5e465302ca32b318489d64b1b2"
server_key_iv: "44c01e3601383ec721e5f0aa7dde38e7"
server_key_version: "encryptionkey@0"
I20260926 02:10:24.276347 997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:24.298648 997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:24.303627 997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:24.303885 997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:24.323525 997 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:32955
I20260926 02:10:24.323545 1064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:32955 every 8 connection(s)
I20260926 02:10:24.325280 997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:10:24.326963 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 997
I20260926 02:10:24.327934 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:40959
--local_ip_for_outbound_sockets=127.18.159.129
--tserver_master_addrs=127.18.159.190:32955
--webserver_port=44783
--webserver_interface=127.18.159.129
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:10:24.333276 1065 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:10:24.339082 1065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977: Bootstrap starting.
I20260926 02:10:24.371237 1065 log.cc:824] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977: Log is configured to *not* fsync() on all Append() calls
I20260926 02:10:24.388705 1065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:10:24.389467 1065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977: Bootstrap complete.
I20260926 02:10:24.405844 1065 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dd73e83ab7481c8743fa65104f9977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32955 } }
I20260926 02:10:24.407254 1065 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dd73e83ab7481c8743fa65104f9977, State: Initialized, Role: FOLLOWER
I20260926 02:10:24.408061 1065 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dd73e83ab7481c8743fa65104f9977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32955 } }
I20260926 02:10:24.408394 1065 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:10:24.408543 1065 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:10:24.408727 1065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:10:24.414734 1065 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dd73e83ab7481c8743fa65104f9977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32955 } }
I20260926 02:10:24.415462 1065 leader_election.cc:302] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0dd73e83ab7481c8743fa65104f9977; no voters:
I20260926 02:10:24.416172 1065 leader_election.cc:288] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 02:10:24.416383 1069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 02:10:24.417950 1069 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 2 LEADER]: Becoming Leader. State: Replica: d0dd73e83ab7481c8743fa65104f9977, State: Running, Role: LEADER
I20260926 02:10:24.418557 1065 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:10:24.418643 1069 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dd73e83ab7481c8743fa65104f9977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32955 } }
I20260926 02:10:24.421226 1070 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d0dd73e83ab7481c8743fa65104f9977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dd73e83ab7481c8743fa65104f9977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32955 } } }
I20260926 02:10:24.421696 1070 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [sys.catalog]: This master's current role is: LEADER
I20260926 02:10:24.421499 1071 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0dd73e83ab7481c8743fa65104f9977. Latest consensus state: current_term: 2 leader_uuid: "d0dd73e83ab7481c8743fa65104f9977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dd73e83ab7481c8743fa65104f9977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32955 } } }
I20260926 02:10:24.422261 1071 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [sys.catalog]: This master's current role is: LEADER
I20260926 02:10:24.423760 1076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:10:24.432912 1076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:10:24.434544 1076 catalog_manager.cc:1259] Loaded cluster ID: a7d169a93dda4a9cb5de71323e415118
I20260926 02:10:24.434731 1076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:10:24.438190 1076 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 02:10:24.457275 1076 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 02:10:24.467607 1076 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:10:24.469600 1076 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977: Loaded TSK: 0
I20260926 02:10:24.471273 1076 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 02:10:24.733985 1067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:24.734349 1067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:24.734524 1067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:24.742527 1067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:24.742718 1067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:24.742916 1067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:10:24.753270 1067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:40959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=44783
--tserver_master_addrs=127.18.159.190:32955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:24.754683 1067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:24.756086 1067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:24.771384 1093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:24.771613 1092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:24.772375 1067 server_base.cc:1034] running on GCE node
W20260926 02:10:24.772389 1095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:24.773252 1067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:24.774487 1067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:24.775754 1067 hybrid_clock.cc:648] HybridClock initialized: now 1790388624775680 us; error 68 us; skew 500 ppm
I20260926 02:10:24.776199 1067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:24 dist-test-slave-03rn krb5kdc[444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388624, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:24.780512 1067 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:10:24.782697 1067 webserver.cc:460] Webserver started at http://127.18.159.129:44783/ using document root <none> and password file <none>
I20260926 02:10:24.783705 1067 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:24.783871 1067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:24.791674 1067 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:10:24.795782 1102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:24.797752 1067 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260926 02:10:24.797961 1067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "8911d79cd5134be8a7cf8838834373d9"
format_stamp: "Formatted at 2026-09-26 02:10:21 on dist-test-slave-03rn"
server_key: "c12269d9153cba41c1d7521fc5a63e15"
server_key_iv: "157d4f2bfe326e5342344ce7e03999fb"
server_key_version: "encryptionkey@0"
I20260926 02:10:24.799459 1067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:24.820246 1067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:24.825554 1067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:24.825917 1067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:24.827111 1067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:10:24.828765 1067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:10:24.828908 1067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:24.829068 1067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:10:24.829181 1067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:24.864442 1067 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:40959
I20260926 02:10:24.864516 1214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:40959 every 8 connection(s)
I20260926 02:10:24.866101 1067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:10:24.867683 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 1067
I20260926 02:10:24.868428 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:38413
--local_ip_for_outbound_sockets=127.18.159.130
--tserver_master_addrs=127.18.159.190:32955
--webserver_port=35661
--webserver_interface=127.18.159.130
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:10:24 dist-test-slave-03rn krb5kdc[444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388624, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:10:24.896322 1218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:24.869824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.129:36375)
0926 02:10:24.870815 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:24.870874 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:24.873825 (+ 2951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:24.875559 (+ 1734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:24.875574 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:24.876340 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:24.876944 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:24.876963 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:24.878699 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:24.878704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:24.880205 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:24.880226 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:24.881503 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:24.881511 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:24.881826 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:24.882603 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:24.882627 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:24.885288 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:24.891319 (+ 6031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:24.891329 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:24.891343 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:24.891910 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:24.892648 (+ 738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:24.892652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:24.892654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:24.892862 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:24.894066 (+ 1204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:24.894073 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:24.894436 (+ 363us) client_negotiation.cc:769] Sending connection context
0926 02:10:24.894683 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 02:10:24.895155 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":403,"threads_started":1}
I20260926 02:10:24.897145 1217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:24.869442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:36375 (local address 127.18.159.190:32955)
0926 02:10:24.870163 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:24.870182 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:24.871058 (+ 876us) server_negotiation.cc:407] Connection header received
0926 02:10:24.874529 (+ 3471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:24.874544 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:24.874959 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:24.875310 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:24.877212 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:24.878498 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:24.880364 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:24.881224 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:24.881474 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:24.885728 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:24.885746 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:24.885774 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:24.885836 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:24.891056 (+ 5220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:24.892123 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:24.892138 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:24.892150 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:24.892486 (+ 336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:24.893014 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:24.893022 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:24.893029 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:24.893897 (+ 868us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:24.894144 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:24.895045 (+ 901us) server_negotiation.cc:299] Negotiation successful
0926 02:10:24.895631 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":295,"threads_started":1}
I20260926 02:10:24.900315 1215 heartbeater.cc:347] Connected to a master server at 127.18.159.190:32955
I20260926 02:10:24.900678 1215 heartbeater.cc:464] Registering TS with master...
I20260926 02:10:24.901472 1215 heartbeater.cc:511] Master 127.18.159.190:32955 requested a full tablet report, sending...
I20260926 02:10:24.903872 1030 ts_manager.cc:194] Registered new tserver with Master: 8911d79cd5134be8a7cf8838834373d9 (127.18.159.129:40959)
I20260926 02:10:24.905735 1030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:36375
W20260926 02:10:25.225838 1219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:25.226150 1219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:25.226321 1219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:25.234223 1219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:25.234342 1219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:25.234540 1219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:10:25.245132 1219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:38413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=35661
--tserver_master_addrs=127.18.159.190:32955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:25.246559 1219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:25.247946 1219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:25.263144 1224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:25.263193 1227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:25.263203 1225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:25.264199 1219 server_base.cc:1034] running on GCE node
I20260926 02:10:25.264892 1219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:25.266055 1219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:25.267295 1219 hybrid_clock.cc:648] HybridClock initialized: now 1790388625267237 us; error 48 us; skew 500 ppm
I20260926 02:10:25.267663 1219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:25 dist-test-slave-03rn krb5kdc[444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388625, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:25.271193 1219 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:10:25.273034 1219 webserver.cc:460] Webserver started at http://127.18.159.130:35661/ using document root <none> and password file <none>
I20260926 02:10:25.273820 1219 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:25.273983 1219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:25.279943 1219 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 02:10:25.283269 1234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:25.285084 1219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:10:25.285293 1219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "9af40ba88bdc4287a6f472bfa69fc076"
format_stamp: "Formatted at 2026-09-26 02:10:22 on dist-test-slave-03rn"
server_key: "379a341e52e62de094db00a685fd168b"
server_key_iv: "b0cf1595c8b5d2a761791633d917429e"
server_key_version: "encryptionkey@0"
I20260926 02:10:25.286593 1219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:25.301028 1219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:25.305467 1219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:25.305787 1219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:25.306790 1219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:10:25.308214 1219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:10:25.308338 1219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:25.308451 1219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:10:25.308532 1219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:25.345345 1219 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:38413
I20260926 02:10:25.345400 1346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:38413 every 8 connection(s)
I20260926 02:10:25.346881 1219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:10:25.356855 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 1219
I20260926 02:10:25.357604 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:35439
--local_ip_for_outbound_sockets=127.18.159.131
--tserver_master_addrs=127.18.159.190:32955
--webserver_port=39045
--webserver_interface=127.18.159.131
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:10:25 dist-test-slave-03rn krb5kdc[444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388625, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:10:25.372108 1217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:25.349600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:59241 (local address 127.18.159.190:32955)
0926 02:10:25.349791 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:25.349796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:25.351072 (+ 1276us) server_negotiation.cc:407] Connection header received
0926 02:10:25.353000 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:25.353005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:25.353124 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:25.353230 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:25.354843 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.355588 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.357914 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.358722 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.358859 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:25.364425 (+ 5566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:25.364435 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:25.364444 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:25.364503 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:25.368331 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:25.369492 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:25.369497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:25.369500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:25.369650 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:25.370180 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:25.370184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:25.370186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:25.370496 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:25.370627 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:25.371629 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0926 02:10:25.371837 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:10:25.373670 1349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:25.350126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.130:59241)
0926 02:10:25.350895 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:25.350933 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:25.352689 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:25.353454 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:25.353463 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:25.354023 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:25.354629 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:25.354648 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.355758 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.355762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:25.357704 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:25.357721 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.359260 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.359269 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:25.359667 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:25.360660 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:25.360698 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:25.364087 (+ 3389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:25.368567 (+ 4480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:25.368582 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:25.368606 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:25.369330 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:25.369771 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:25.369779 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:25.369786 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:25.370044 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:25.370638 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:25.370649 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:25.371146 (+ 497us) client_negotiation.cc:769] Sending connection context
0926 02:10:25.371501 (+ 355us) client_negotiation.cc:240] Negotiation successful
0926 02:10:25.372111 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":345,"threads_started":1}
I20260926 02:10:25.375349 1347 heartbeater.cc:347] Connected to a master server at 127.18.159.190:32955
I20260926 02:10:25.375692 1347 heartbeater.cc:464] Registering TS with master...
I20260926 02:10:25.376541 1347 heartbeater.cc:511] Master 127.18.159.190:32955 requested a full tablet report, sending...
I20260926 02:10:25.378149 1030 ts_manager.cc:194] Registered new tserver with Master: 9af40ba88bdc4287a6f472bfa69fc076 (127.18.159.130:38413)
I20260926 02:10:25.379539 1030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:59241
W20260926 02:10:25.723397 1350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:25.723719 1350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:25.723815 1350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:25.731778 1350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:25.731905 1350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:25.732023 1350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:10:25.742923 1350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:35439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=39045
--tserver_master_addrs=127.18.159.190:32955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:25.744282 1350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:25.745649 1350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:25.759821 1356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:25.759963 1358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:25.760071 1355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:25.760865 1350 server_base.cc:1034] running on GCE node
I20260926 02:10:25.761608 1350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:25.762732 1350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:25.764019 1350 hybrid_clock.cc:648] HybridClock initialized: now 1790388625763948 us; error 54 us; skew 500 ppm
I20260926 02:10:25.764459 1350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:25 dist-test-slave-03rn krb5kdc[444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388625, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:25.768285 1350 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:10:25.770097 1350 webserver.cc:460] Webserver started at http://127.18.159.131:39045/ using document root <none> and password file <none>
I20260926 02:10:25.770896 1350 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:25.771078 1350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:25.777326 1350 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:10:25.780645 1365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:25.782501 1350 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260926 02:10:25.782729 1350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "2601425df57849a6bbc056dcbf1ea73d"
format_stamp: "Formatted at 2026-09-26 02:10:22 on dist-test-slave-03rn"
server_key: "49454775f30aca5cf742b98b19d53ef4"
server_key_iv: "3115ddd3aef163aba5bba9d16edeb9cb"
server_key_version: "encryptionkey@0"
I20260926 02:10:25.783763 1350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:25.807150 1350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:25.811957 1350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:25.812247 1350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:25.813241 1350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:10:25.814550 1350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:10:25.814639 1350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:25.814715 1350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:10:25.814769 1350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:25.848784 1350 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:35439
I20260926 02:10:25.848834 1477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:35439 every 8 connection(s)
I20260926 02:10:25.850275 1350 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:10:25.859093 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 1350
Sep 26 02:10:25 dist-test-slave-03rn krb5kdc[444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388625, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:10:25.879798 1217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:25.852916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:34285 (local address 127.18.159.190:32955)
0926 02:10:25.853155 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:25.853161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:25.854685 (+ 1524us) server_negotiation.cc:407] Connection header received
0926 02:10:25.858019 (+ 3334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:25.858030 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:25.858184 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:25.858308 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:25.861342 (+ 3034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.862095 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.864495 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.865204 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.865302 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:25.872042 (+ 6740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:25.872052 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:25.872059 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:25.872109 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:25.875519 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:25.876532 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:25.876538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:25.876542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:25.876682 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:25.877268 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:25.877274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:25.877280 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:25.877684 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:25.877842 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:25.879062 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0926 02:10:25.879368 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 02:10:25.881314 1480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:25.853437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.131:34285)
0926 02:10:25.854433 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:25.854494 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:25.857517 (+ 3023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:25.858703 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:25.858765 (+ 62us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:25.859647 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:25.861121 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:25.861152 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.862263 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.862270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:25.864286 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:25.864303 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.865644 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.865650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:25.865966 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:25.866945 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:25.867001 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:25.871733 (+ 4732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:25.875732 (+ 3999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:25.875741 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:25.875755 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:25.876354 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:25.876810 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:25.876816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:25.876820 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:25.877106 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:25.877964 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:25.877989 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:25.878542 (+ 553us) client_negotiation.cc:769] Sending connection context
0926 02:10:25.878886 (+ 344us) client_negotiation.cc:240] Negotiation successful
0926 02:10:25.879572 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":425,"threads_started":1}
I20260926 02:10:25.885228 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:25.860209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52838 (local address 127.18.159.190:32955)
0926 02:10:25.860697 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:25.860703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:25.864263 (+ 3560us) server_negotiation.cc:407] Connection header received
0926 02:10:25.864340 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:25.864348 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:25.864439 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:25.864584 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:25.866649 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.867778 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.870744 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.871445 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.871610 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:25.878082 (+ 6472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:25.878094 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:25.878097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:25.878128 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:25.881712 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:25.882696 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:25.882702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:25.882706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:25.882895 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:25.883430 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:25.883435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:25.883438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:25.883885 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:25.884148 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:25.884611 (+ 463us) server_negotiation.cc:299] Negotiation successful
0926 02:10:25.884853 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":252,"threads_started":1}
I20260926 02:10:25.885417 1478 heartbeater.cc:347] Connected to a master server at 127.18.159.190:32955
I20260926 02:10:25.886202 1478 heartbeater.cc:464] Registering TS with master...
I20260926 02:10:25.887214 1478 heartbeater.cc:511] Master 127.18.159.190:32955 requested a full tablet report, sending...
I20260926 02:10:25.888916 1030 ts_manager.cc:194] Registered new tserver with Master: 2601425df57849a6bbc056dcbf1ea73d (127.18.159.131:35439)
I20260926 02:10:25.889935 1030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:34285
I20260926 02:10:25.901126 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:10:25.909163 1215 heartbeater.cc:503] Master 127.18.159.190:32955 was elected leader, sending a full tablet report...
I20260926 02:10:25.910303 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:25.902249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52854 (local address 127.18.159.190:32955)
0926 02:10:25.902510 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:25.902520 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:25.903078 (+ 558us) server_negotiation.cc:407] Connection header received
0926 02:10:25.903342 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:25.903346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:25.903538 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:25.903689 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:10:25.904350 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.905150 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.907703 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.908240 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.908351 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:25.908920 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:10:25.909601 (+ 681us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:10:25.909768 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:25.909899 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 02:10:25.909982 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:10:25.911440 1029 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:52854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:10:25.919013 1029 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:10:25.947402 1489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:25.931758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36242 (local address 127.18.159.131:35439)
0926 02:10:25.936639 (+ 4881us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:25.936650 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:25.937574 (+ 924us) server_negotiation.cc:407] Connection header received
0926 02:10:25.938005 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:25.938014 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:25.938280 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:25.938449 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:10:25.939640 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.941357 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.944242 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.945765 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.945941 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:25.946153 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:25.946854 (+ 701us) server_negotiation.cc:299] Negotiation successful
0926 02:10:25.947042 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4708,"thread_start_us":4558,"threads_started":1}
I20260926 02:10:25.948436 1488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:25.931627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:35439 (local address 127.0.0.1:36242)
0926 02:10:25.937395 (+ 5768us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:25.937453 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:25.937742 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:25.938842 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:25.938850 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:25.938891 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:10:25.939377 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:25.939415 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.941546 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.941552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:25.944032 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:25.944049 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.946344 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.946351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:25.946577 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:25.946636 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 02:10:25.947723 (+ 1087us) client_negotiation.cc:240] Negotiation successful
0926 02:10:25.947933 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5451,"spinlock_wait_cycles":10240,"thread_start_us":5243,"threads_started":1}
I20260926 02:10:25.950381 1491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:25.932587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34918 (local address 127.18.159.130:38413)
0926 02:10:25.933331 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:25.933344 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:25.937577 (+ 4233us) server_negotiation.cc:407] Connection header received
0926 02:10:25.938008 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:25.938015 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:25.938280 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:25.938449 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:10:25.940713 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.942266 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.946131 (+ 3865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.947617 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.947756 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:25.947916 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:25.949856 (+ 1940us) server_negotiation.cc:299] Negotiation successful
0926 02:10:25.950066 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":365,"threads_started":1}
I20260926 02:10:25.951198 1490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:25.931903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:38413 (local address 127.0.0.1:34918)
0926 02:10:25.937395 (+ 5492us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:25.937453 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:25.937742 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:25.940064 (+ 2322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:25.940071 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:25.940096 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:10:25.940507 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:25.940519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.942491 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.942496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:25.945934 (+ 3438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:25.945947 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.948147 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.948175 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:25.949625 (+ 1450us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:25.949662 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 02:10:25.950588 (+ 926us) client_negotiation.cc:240] Negotiation successful
0926 02:10:25.950746 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5233,"spinlock_wait_cycles":12544,"thread_start_us":4993,"threads_started":1}
I20260926 02:10:25.952858 1413 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd6eaa7952247dcb2355427ace7c56d (DEFAULT_TABLE table=test-table [id=78144376b24142e78eaacf20685e88ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:10:25.954487 1413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd6eaa7952247dcb2355427ace7c56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:10:25.954918 1282 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd6eaa7952247dcb2355427ace7c56d (DEFAULT_TABLE table=test-table [id=78144376b24142e78eaacf20685e88ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:10:25.956254 1282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd6eaa7952247dcb2355427ace7c56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:10:25.960196 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:25.937902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40959 (local address 127.0.0.1:60840)
0926 02:10:25.949589 (+ 11687us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:25.949604 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:25.949779 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:25.950414 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:25.950419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:25.950737 (+ 318us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:10:25.951049 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:25.951058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.953596 (+ 2538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.953602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:25.957063 (+ 3461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:25.957085 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.959239 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.959247 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:25.959469 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:25.959511 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 02:10:25.959622 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 02:10:25.959785 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11582,"thread_start_us":11992,"threads_started":1}
I20260926 02:10:25.960227 1493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:25.938292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.18.159.129:40959)
0926 02:10:25.949189 (+ 10897us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:25.949196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:25.949736 (+ 540us) server_negotiation.cc:407] Connection header received
0926 02:10:25.949964 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:25.949972 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:25.950220 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:25.950447 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:10:25.951229 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.952660 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.957307 (+ 4647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:25.958924 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:25.959168 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:25.959365 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:25.959684 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 02:10:25.959872 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10717,"thread_start_us":10561,"threads_started":1}
I20260926 02:10:25.964393 1150 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd6eaa7952247dcb2355427ace7c56d (DEFAULT_TABLE table=test-table [id=78144376b24142e78eaacf20685e88ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:10:25.965766 1150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd6eaa7952247dcb2355427ace7c56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:10:25.969864 1494 tablet_bootstrap.cc:492] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d: Bootstrap starting.
I20260926 02:10:25.974457 1494 tablet_bootstrap.cc:654] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d: Neither blocks nor log segments found. Creating new log.
I20260926 02:10:25.974997 1495 tablet_bootstrap.cc:492] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076: Bootstrap starting.
I20260926 02:10:25.975804 1494 log.cc:824] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:10:25.978564 1494 tablet_bootstrap.cc:492] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d: No bootstrap required, opened a new log
I20260926 02:10:25.978814 1495 tablet_bootstrap.cc:654] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076: Neither blocks nor log segments found. Creating new log.
I20260926 02:10:25.978854 1494 ts_tablet_manager.cc:1397] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 02:10:25.980271 1495 log.cc:824] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076: Log is configured to *not* fsync() on all Append() calls
I20260926 02:10:25.980577 1497 tablet_bootstrap.cc:492] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9: Bootstrap starting.
I20260926 02:10:25.984634 1497 tablet_bootstrap.cc:654] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9: Neither blocks nor log segments found. Creating new log.
I20260926 02:10:25.986168 1497 log.cc:824] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9: Log is configured to *not* fsync() on all Append() calls
I20260926 02:10:25.989534 1497 tablet_bootstrap.cc:492] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9: No bootstrap required, opened a new log
I20260926 02:10:25.989617 1495 tablet_bootstrap.cc:492] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076: No bootstrap required, opened a new log
I20260926 02:10:25.989876 1497 ts_tablet_manager.cc:1397] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260926 02:10:25.989902 1495 ts_tablet_manager.cc:1397] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260926 02:10:25.990132 1494 raft_consensus.cc:348] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2601425df57849a6bbc056dcbf1ea73d" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35439 } } peers { permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 } } peers { permanent_uuid: "8911d79cd5134be8a7cf8838834373d9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40959 } }
I20260926 02:10:25.990660 1494 raft_consensus.cc:374] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:10:25.990772 1494 raft_consensus.cc:729] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2601425df57849a6bbc056dcbf1ea73d, State: Initialized, Role: FOLLOWER
I20260926 02:10:25.991442 1494 consensus_queue.cc:260] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2601425df57849a6bbc056dcbf1ea73d" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35439 } } peers { permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 } } peers { permanent_uuid: "8911d79cd5134be8a7cf8838834373d9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40959 } }
I20260926 02:10:25.992678 1478 heartbeater.cc:503] Master 127.18.159.190:32955 was elected leader, sending a full tablet report...
I20260926 02:10:25.993471 1494 ts_tablet_manager.cc:1428] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260926 02:10:26.000290 1495 raft_consensus.cc:348] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2601425df57849a6bbc056dcbf1ea73d" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35439 } } peers { permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 } } peers { permanent_uuid: "8911d79cd5134be8a7cf8838834373d9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40959 } }
I20260926 02:10:26.000716 1495 raft_consensus.cc:374] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:10:26.000837 1495 raft_consensus.cc:729] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9af40ba88bdc4287a6f472bfa69fc076, State: Initialized, Role: FOLLOWER
I20260926 02:10:26.001438 1495 consensus_queue.cc:260] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2601425df57849a6bbc056dcbf1ea73d" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35439 } } peers { permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 } } peers { permanent_uuid: "8911d79cd5134be8a7cf8838834373d9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40959 } }
I20260926 02:10:26.001700 1497 raft_consensus.cc:348] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2601425df57849a6bbc056dcbf1ea73d" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35439 } } peers { permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 } } peers { permanent_uuid: "8911d79cd5134be8a7cf8838834373d9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40959 } }
I20260926 02:10:26.002187 1497 raft_consensus.cc:374] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:10:26.002295 1497 raft_consensus.cc:729] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8911d79cd5134be8a7cf8838834373d9, State: Initialized, Role: FOLLOWER
I20260926 02:10:26.002592 1347 heartbeater.cc:503] Master 127.18.159.190:32955 was elected leader, sending a full tablet report...
I20260926 02:10:26.002928 1497 consensus_queue.cc:260] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2601425df57849a6bbc056dcbf1ea73d" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35439 } } peers { permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 } } peers { permanent_uuid: "8911d79cd5134be8a7cf8838834373d9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40959 } }
I20260926 02:10:26.003314 1495 ts_tablet_manager.cc:1428] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260926 02:10:26.004911 1497 ts_tablet_manager.cc:1428] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260926 02:10:26.049609 1502 raft_consensus.cc:482] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:10:26.049966 1502 raft_consensus.cc:504] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2601425df57849a6bbc056dcbf1ea73d" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35439 } } peers { permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 } } peers { permanent_uuid: "8911d79cd5134be8a7cf8838834373d9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40959 } }
I20260926 02:10:26.051824 1502 leader_election.cc:288] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2601425df57849a6bbc056dcbf1ea73d (127.18.159.131:35439), 9af40ba88bdc4287a6f472bfa69fc076 (127.18.159.130:38413)
I20260926 02:10:26.059485 1491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:26.052338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:51333 (local address 127.18.159.130:38413)
0926 02:10:26.052560 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:26.052568 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:26.053351 (+ 783us) server_negotiation.cc:407] Connection header received
0926 02:10:26.053539 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:26.053542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:26.053610 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:26.053727 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:10:26.054397 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:26.055106 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:26.057631 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:26.058431 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:26.058625 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:26.058706 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:26.059059 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 02:10:26.059191 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:10:26.059620 1489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:26.052321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:36153 (local address 127.18.159.131:35439)
0926 02:10:26.052622 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:26.052629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:26.053292 (+ 663us) server_negotiation.cc:407] Connection header received
0926 02:10:26.053504 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:26.053510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:26.053579 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:26.053647 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:10:26.054334 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:26.055102 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:26.057556 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:26.058463 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:26.058637 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:26.058716 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:26.059201 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 02:10:26.059334 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:10:26.059755 1504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:26.052129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:35439 (local address 127.18.159.129:36153)
0926 02:10:26.053153 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:26.053169 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:26.053338 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:26.053699 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:26.053708 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:26.053734 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:10:26.054174 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:26.054187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:26.055246 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:26.055255 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:26.057380 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:26.057403 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:26.058848 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:26.058852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:26.059013 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:26.059058 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 02:10:26.059170 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 02:10:26.059310 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":44,"thread_start_us":714,"threads_started":1}
I20260926 02:10:26.060148 1503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:26.052338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:38413 (local address 127.18.159.129:51333)
0926 02:10:26.053193 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:26.053208 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:26.053377 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:26.053765 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:26.053770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:26.053802 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:10:26.054212 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:26.054222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:26.055255 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:26.055261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:26.057386 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:26.057403 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:26.058663 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:26.058669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:26.058846 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:26.058872 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 02:10:26.059017 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 02:10:26.059164 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":58,"thread_start_us":432,"threads_started":1}
I20260926 02:10:26.060873 1433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd6eaa7952247dcb2355427ace7c56d" candidate_uuid: "8911d79cd5134be8a7cf8838834373d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2601425df57849a6bbc056dcbf1ea73d" is_pre_election: true
I20260926 02:10:26.061137 1302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd6eaa7952247dcb2355427ace7c56d" candidate_uuid: "8911d79cd5134be8a7cf8838834373d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9af40ba88bdc4287a6f472bfa69fc076" is_pre_election: true
I20260926 02:10:26.061424 1433 raft_consensus.cc:2454] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8911d79cd5134be8a7cf8838834373d9 in term 0.
I20260926 02:10:26.061703 1302 raft_consensus.cc:2454] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8911d79cd5134be8a7cf8838834373d9 in term 0.
I20260926 02:10:26.062163 1103 leader_election.cc:302] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2601425df57849a6bbc056dcbf1ea73d, 8911d79cd5134be8a7cf8838834373d9; no voters:
I20260926 02:10:26.062609 1502 raft_consensus.cc:2781] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:10:26.062780 1502 raft_consensus.cc:482] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:10:26.062891 1502 raft_consensus.cc:3037] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:10:26.065232 1502 raft_consensus.cc:504] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2601425df57849a6bbc056dcbf1ea73d" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35439 } } peers { permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 } } peers { permanent_uuid: "8911d79cd5134be8a7cf8838834373d9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40959 } }
I20260926 02:10:26.065876 1502 leader_election.cc:288] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 [CANDIDATE]: Term 1 election: Requested vote from peers 2601425df57849a6bbc056dcbf1ea73d (127.18.159.131:35439), 9af40ba88bdc4287a6f472bfa69fc076 (127.18.159.130:38413)
I20260926 02:10:26.066435 1433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd6eaa7952247dcb2355427ace7c56d" candidate_uuid: "8911d79cd5134be8a7cf8838834373d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2601425df57849a6bbc056dcbf1ea73d"
I20260926 02:10:26.066417 1302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd6eaa7952247dcb2355427ace7c56d" candidate_uuid: "8911d79cd5134be8a7cf8838834373d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9af40ba88bdc4287a6f472bfa69fc076"
I20260926 02:10:26.066655 1302 raft_consensus.cc:3037] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:10:26.066655 1433 raft_consensus.cc:3037] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:10:26.070068 1302 raft_consensus.cc:2454] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8911d79cd5134be8a7cf8838834373d9 in term 1.
I20260926 02:10:26.070068 1433 raft_consensus.cc:2454] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8911d79cd5134be8a7cf8838834373d9 in term 1.
I20260926 02:10:26.070601 1103 leader_election.cc:302] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2601425df57849a6bbc056dcbf1ea73d, 8911d79cd5134be8a7cf8838834373d9; no voters:
I20260926 02:10:26.070986 1502 raft_consensus.cc:2781] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:10:26.071453 1502 raft_consensus.cc:686] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 [term 1 LEADER]: Becoming Leader. State: Replica: 8911d79cd5134be8a7cf8838834373d9, State: Running, Role: LEADER
I20260926 02:10:26.071964 1502 consensus_queue.cc:237] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2601425df57849a6bbc056dcbf1ea73d" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35439 } } peers { permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 } } peers { permanent_uuid: "8911d79cd5134be8a7cf8838834373d9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40959 } }
I20260926 02:10:26.076721 1030 catalog_manager.cc:5495] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8911d79cd5134be8a7cf8838834373d9 (127.18.159.129). New cstate: current_term: 1 leader_uuid: "8911d79cd5134be8a7cf8838834373d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2601425df57849a6bbc056dcbf1ea73d" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8911d79cd5134be8a7cf8838834373d9" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40959 } health_report { overall_health: HEALTHY } } }
W20260926 02:10:26.099067 1348 tablet.cc:2369] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:10:26.101691 1493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:26.095361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60848 (local address 127.18.159.129:40959)
0926 02:10:26.095585 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:26.095590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:26.095668 (+ 78us) server_negotiation.cc:407] Connection header received
0926 02:10:26.095919 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:26.095924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:26.096056 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:26.096306 (+ 250us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:10:26.096748 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:26.097504 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:26.099039 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:26.099560 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:26.099664 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:26.100401 (+ 737us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:10:26.100986 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:10:26.101168 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:26.101259 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 02:10:26.101341 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 02:10:26.101946 1479 tablet.cc:2369] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:10:26.111050 1302 raft_consensus.cc:1261] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076 [term 1 FOLLOWER]: Refusing update from remote peer 8911d79cd5134be8a7cf8838834373d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:10:26.111092 1433 raft_consensus.cc:1261] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d [term 1 FOLLOWER]: Refusing update from remote peer 8911d79cd5134be8a7cf8838834373d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:10:26.112221 1505 consensus_queue.cc:1035] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2601425df57849a6bbc056dcbf1ea73d" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:10:26.112603 1502 consensus_queue.cc:1035] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9af40ba88bdc4287a6f472bfa69fc076" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 38413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 02:10:26.119046 1216 tablet.cc:2369] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:10:26.125295 1508 mvcc.cc:205] Tried to move back new op lower bound from 7333431812540293120 to 7333431812395032576. Current Snapshot: MvccSnapshot[applied={T|T < 7333431812540293120}]
I20260926 02:10:26.130092 1510 mvcc.cc:205] Tried to move back new op lower bound from 7333431812540293120 to 7333431812395032576. Current Snapshot: MvccSnapshot[applied={T|T < 7333431812540293120}]
I20260926 02:10:26.145190 1028 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:52854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:10:26.145552 1028 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:52854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:10:26.149910 1028 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977: Sending DeleteTablet for 3 replicas of tablet 3fd6eaa7952247dcb2355427ace7c56d
I20260926 02:10:26.151258 1413 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd6eaa7952247dcb2355427ace7c56d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:10:26 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:36242
I20260926 02:10:26.151361 1282 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd6eaa7952247dcb2355427ace7c56d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:10:26 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:34918
I20260926 02:10:26.151499 1150 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd6eaa7952247dcb2355427ace7c56d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:10:26 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:60840
I20260926 02:10:26.152531 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 1067
I20260926 02:10:26.162750 1518 tablet_replica.cc:330] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076: stopping tablet replica
I20260926 02:10:26.163002 1519 tablet_replica.cc:330] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d: stopping tablet replica
I20260926 02:10:26.163450 1519 raft_consensus.cc:2229] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:10:26.163453 1518 raft_consensus.cc:2229] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:10:26.164058 1519 raft_consensus.cc:2258] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:10:26.164048 1518 raft_consensus.cc:2258] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:10:26.164958 1520 tablet_replica.cc:330] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9: stopping tablet replica
I20260926 02:10:26.165463 1520 raft_consensus.cc:2229] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:10:26.166240 1520 raft_consensus.cc:2258] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:10:26.166917 1519 ts_tablet_manager.cc:1905] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:10:26.168777 1520 ts_tablet_manager.cc:1905] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:10:26.170636 1518 ts_tablet_manager.cc:1905] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:10:26.175288 1493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:26.154016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60854 (local address 127.18.159.129:40959)
0926 02:10:26.154205 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:26.154212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:26.154247 (+ 35us) server_negotiation.cc:407] Connection header received
0926 02:10:26.154307 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:26.154312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:26.154397 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:26.154494 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:26.156619 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:26.157851 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:26.159895 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:26.160663 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:26.160789 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:26.165750 (+ 4961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:26.165767 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:26.165786 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:26.165844 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:26.170634 (+ 4790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:26.171716 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:26.171725 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:26.171729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:26.171907 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:26.172436 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:26.172441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:26.172445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:26.172923 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:26.174451 (+ 1528us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:26.174635 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 02:10:26.174901 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:10:26.176096 1210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:26.177716 1518 ts_tablet_manager.cc:1918] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:10:26.177930 1518 log.cc:1196] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/3fd6eaa7952247dcb2355427ace7c56d
I20260926 02:10:26.178413 1518 ts_tablet_manager.cc:1939] T 3fd6eaa7952247dcb2355427ace7c56d P 9af40ba88bdc4287a6f472bfa69fc076: Deleting consensus metadata
I20260926 02:10:26.180233 1015 catalog_manager.cc:4841] TS 9af40ba88bdc4287a6f472bfa69fc076 (127.18.159.130:38413): tablet 3fd6eaa7952247dcb2355427ace7c56d (table test-table [id=78144376b24142e78eaacf20685e88ff]) successfully deleted
I20260926 02:10:26.184628 1519 ts_tablet_manager.cc:1918] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:10:26.184835 1519 log.cc:1196] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/3fd6eaa7952247dcb2355427ace7c56d
I20260926 02:10:26.185310 1519 ts_tablet_manager.cc:1939] T 3fd6eaa7952247dcb2355427ace7c56d P 2601425df57849a6bbc056dcbf1ea73d: Deleting consensus metadata
I20260926 02:10:26.186950 1014 catalog_manager.cc:4841] TS 2601425df57849a6bbc056dcbf1ea73d (127.18.159.131:35439): tablet 3fd6eaa7952247dcb2355427ace7c56d (table test-table [id=78144376b24142e78eaacf20685e88ff]) successfully deleted
I20260926 02:10:26.278702 1520 ts_tablet_manager.cc:1918] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:10:26.279014 1520 log.cc:1196] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/3fd6eaa7952247dcb2355427ace7c56d
I20260926 02:10:26.279628 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 1067
I20260926 02:10:26.298318 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 1219
W20260926 02:10:26.298416 1015 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.159.129:40959 (error 108)
I20260926 02:10:26.300328 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:26.299846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40959 (local address 127.0.0.1:60868)
0926 02:10:26.300007 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:26.300126 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 02:10:26.300594 1015 catalog_manager.cc:4584] TS 8911d79cd5134be8a7cf8838834373d9 (127.18.159.129:40959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3fd6eaa7952247dcb2355427ace7c56d: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40959: connect: Connection refused (error 111)
I20260926 02:10:26.313090 1491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:26.299223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34934 (local address 127.18.159.130:38413)
0926 02:10:26.299449 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:26.299456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:26.299547 (+ 91us) server_negotiation.cc:407] Connection header received
0926 02:10:26.299772 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:26.299777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:26.299843 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:26.299949 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:26.301554 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:26.302299 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:26.303692 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:26.304079 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:26.304266 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:26.306916 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:26.306932 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:26.306948 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:26.306991 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:26.309717 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:26.310496 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:26.310504 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:26.310507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:26.310648 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:26.311043 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:26.311049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:26.311053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:26.311469 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:26.311782 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:26.312103 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 02:10:26.312667 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:10:26.313833 1342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:26.354147 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:26.353628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40959 (local address 127.0.0.1:60870)
0926 02:10:26.353841 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:26.353957 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 02:10:26.355417 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:26.354950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40959 (local address 127.0.0.1:60872)
0926 02:10:26.355126 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:26.355227 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 02:10:26.409359 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 1219
I20260926 02:10:26.416258 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:26.415553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40959 (local address 127.0.0.1:60880)
0926 02:10:26.415859 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:26.415983 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260926 02:10:26.417449 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:26.416995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40959 (local address 127.0.0.1:60890)
0926 02:10:26.417184 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:26.417283 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 02:10:26.427666 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 1350
I20260926 02:10:26.442381 1489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:26.428459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36248 (local address 127.18.159.131:35439)
0926 02:10:26.428656 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:26.428662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:26.428710 (+ 48us) server_negotiation.cc:407] Connection header received
0926 02:10:26.428946 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:26.428950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:26.429013 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:26.429122 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:26.430505 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:26.431240 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:26.432835 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:26.433266 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:26.433407 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:26.436044 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:26.436060 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:26.436075 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:26.436117 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:26.439303 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:26.440132 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:26.440141 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:26.440143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:26.440367 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:26.440762 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:26.440768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:26.440771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:26.441142 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:26.441476 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:26.441800 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 02:10:26.442047 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:10:26.443107 1473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:26.498425 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:26.497878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40959 (local address 127.0.0.1:60902)
0926 02:10:26.498124 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:26.498247 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 02:10:26.499603 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:26.499184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40959 (local address 127.0.0.1:60906)
0926 02:10:26.499407 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:26.499482 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 02:10:26.534400 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 1350
I20260926 02:10:26.552610 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 997
I20260926 02:10:26.553786 1060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:26.669502 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 997
I20260926 02:10:26.693053 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:32955
--webserver_interface=127.18.159.190
--webserver_port=44401
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:32955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:10:27.044770 1526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:27.045102 1526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:27.045197 1526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:27.052631 1526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:10:27.052757 1526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:27.052805 1526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:27.052866 1526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:10:27.052917 1526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:10:27.063277 1526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:32955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:32955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=44401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:27.064786 1526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:27.066087 1526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:27.078271 1531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:27.078238 1532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:27.078181 1534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:27.079564 1526 server_base.cc:1034] running on GCE node
I20260926 02:10:27.080545 1526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:27.082116 1526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:27.083379 1526 hybrid_clock.cc:648] HybridClock initialized: now 1790388627083303 us; error 59 us; skew 500 ppm
I20260926 02:10:27.083786 1526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:27 dist-test-slave-03rn krb5kdc[444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388627, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:27.087144 1526 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:10:27.088809 1526 webserver.cc:460] Webserver started at http://127.18.159.190:44401/ using document root <none> and password file <none>
I20260926 02:10:27.089545 1526 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:27.089679 1526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:27.095863 1526 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 02:10:27.099490 1541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:27.101234 1526 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:10:27.101497 1526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "d0dd73e83ab7481c8743fa65104f9977"
format_stamp: "Formatted at 2026-09-26 02:10:20 on dist-test-slave-03rn"
server_key: "20c63c5e465302ca32b318489d64b1b2"
server_key_iv: "44c01e3601383ec721e5f0aa7dde38e7"
server_key_version: "encryptionkey@0"
I20260926 02:10:27.102555 1526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:27.132277 1526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:27.136791 1526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:27.137048 1526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:27.155768 1526 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:32955
I20260926 02:10:27.155777 1592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:32955 every 8 connection(s)
I20260926 02:10:27.157433 1526 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:10:27.158496 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 1526
I20260926 02:10:27.159329 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:40959
--local_ip_for_outbound_sockets=127.18.159.129
--tserver_master_addrs=127.18.159.190:32955
--webserver_port=44783
--webserver_interface=127.18.159.129
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:10:27.165190 1593 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:10:27.169263 1593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977: Bootstrap starting.
I20260926 02:10:27.199167 1593 log.cc:824] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977: Log is configured to *not* fsync() on all Append() calls
I20260926 02:10:27.216761 1593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:10:27.217370 1593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977: Bootstrap complete.
I20260926 02:10:27.226909 1593 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dd73e83ab7481c8743fa65104f9977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32955 } }
I20260926 02:10:27.227907 1593 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dd73e83ab7481c8743fa65104f9977, State: Initialized, Role: FOLLOWER
I20260926 02:10:27.228662 1593 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dd73e83ab7481c8743fa65104f9977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32955 } }
I20260926 02:10:27.228919 1593 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:10:27.229049 1593 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:10:27.229249 1593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 2 FOLLOWER]: Advancing to term 3
I20260926 02:10:27.233345 1593 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dd73e83ab7481c8743fa65104f9977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32955 } }
I20260926 02:10:27.233904 1593 leader_election.cc:302] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0dd73e83ab7481c8743fa65104f9977; no voters:
I20260926 02:10:27.234653 1593 leader_election.cc:288] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 02:10:27.234807 1597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 02:10:27.235970 1597 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [term 3 LEADER]: Becoming Leader. State: Replica: d0dd73e83ab7481c8743fa65104f9977, State: Running, Role: LEADER
I20260926 02:10:27.236624 1597 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dd73e83ab7481c8743fa65104f9977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32955 } }
I20260926 02:10:27.237035 1593 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:10:27.239617 1598 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d0dd73e83ab7481c8743fa65104f9977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dd73e83ab7481c8743fa65104f9977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32955 } } }
I20260926 02:10:27.240099 1598 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [sys.catalog]: This master's current role is: LEADER
I20260926 02:10:27.240453 1599 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0dd73e83ab7481c8743fa65104f9977. Latest consensus state: current_term: 3 leader_uuid: "d0dd73e83ab7481c8743fa65104f9977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dd73e83ab7481c8743fa65104f9977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32955 } } }
I20260926 02:10:27.241009 1599 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dd73e83ab7481c8743fa65104f9977 [sys.catalog]: This master's current role is: LEADER
I20260926 02:10:27.248946 1602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:10:27.258430 1602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:10:27.260371 1602 catalog_manager.cc:1259] Loaded cluster ID: a7d169a93dda4a9cb5de71323e415118
I20260926 02:10:27.260524 1602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 02:10:27.262842 1602 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f77c99cc9ad google::LogMessage::Fail() at ??:0
@ 0x7f77c99cf88a google::LogMessage::SendToLog() at ??:0
@ 0x7f77c99cc4de google::LogMessage::Flush() at ??:0
@ 0x7f77c99cda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f77df3ff512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f77df3fc976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f77df484610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f77df484439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f77cc618345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f77cc61b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f77cc61b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f77cc5f6eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f77cf8d06db start_thread at ??:0
@ 0x7f77c886661f clone at ??:0
W20260926 02:10:27.565212 1595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:27.565605 1595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:27.565779 1595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:27.573576 1595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:27.573755 1595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:27.573930 1595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:10:27.584496 1595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:40959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=44783
--tserver_master_addrs=127.18.159.190:32955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:27.585889 1595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:27.587253 1595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:27.601264 1634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:27.601244 1635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:27.601162 1637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:27.602057 1595 server_base.cc:1034] running on GCE node
I20260926 02:10:27.602733 1595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:27.603785 1595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:27.605074 1595 hybrid_clock.cc:648] HybridClock initialized: now 1790388627604987 us; error 40 us; skew 500 ppm
I20260926 02:10:27.605441 1595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:27 dist-test-slave-03rn krb5kdc[444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388627, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:27.608742 1595 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:10:27.610301 1595 webserver.cc:460] Webserver started at http://127.18.159.129:44783/ using document root <none> and password file <none>
I20260926 02:10:27.611037 1595 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:27.611189 1595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:27.617036 1595 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:10:27.620059 1644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:27.621548 1595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:10:27.621757 1595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "8911d79cd5134be8a7cf8838834373d9"
format_stamp: "Formatted at 2026-09-26 02:10:21 on dist-test-slave-03rn"
server_key: "c12269d9153cba41c1d7521fc5a63e15"
server_key_iv: "157d4f2bfe326e5342344ce7e03999fb"
server_key_version: "encryptionkey@0"
I20260926 02:10:27.622814 1595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:27.638628 1595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:27.643129 1595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:27.643450 1595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:27.644667 1595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:10:27.646603 1651 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260926 02:10:27.652256 1651 ts_tablet_manager.cc:1852] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 02:10:27.653275 1651 ts_tablet_manager.cc:1905] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:10:27.661147 1651 ts_tablet_manager.cc:1918] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:10:27.661507 1651 ts_tablet_manager.cc:1939] T 3fd6eaa7952247dcb2355427ace7c56d P 8911d79cd5134be8a7cf8838834373d9: Deleting consensus metadata
I20260926 02:10:27.661999 1595 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 02:10:27.662173 1595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260926 02:10:27.662338 1595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:10:27.662436 1595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:27.698478 1595 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:40959
I20260926 02:10:27.698529 1757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:40959 every 8 connection(s)
I20260926 02:10:27.699973 1595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:10:27.705621 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 1595
I20260926 02:10:27.706041 1760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:27.703596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.129:58403)
0926 02:10:27.704461 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:27.704726 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":318,"threads_started":1}
I20260926 02:10:27.706418 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:38413
--local_ip_for_outbound_sockets=127.18.159.130
--tserver_master_addrs=127.18.159.190:32955
--webserver_port=35661
--webserver_interface=127.18.159.130
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:10:27.707018 1758 heartbeater.cc:644] Failed to heartbeat to 127.18.159.190:32955 (0 consecutive failures): Network error: Failed to ping master at 127.18.159.190:32955: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
I20260926 02:10:27.708140 1760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:27.707710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.129:45799)
0926 02:10:27.707911 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:27.707991 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 02:10:27.709309 1760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:27.708949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.129:34123)
0926 02:10:27.709096 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:27.709175 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 02:10:27.709626 1758 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:10:28.049504 1761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:28.049815 1761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:28.049994 1761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:28.057353 1761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:28.057492 1761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:28.057693 1761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:10:28.067629 1761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:38413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=35661
--tserver_master_addrs=127.18.159.190:32955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:28.069033 1761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:28.070397 1761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:28.084146 1769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:28.084241 1767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:28.084241 1766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:28.084791 1761 server_base.cc:1034] running on GCE node
I20260926 02:10:28.085530 1761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:28.086526 1761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:28.087766 1761 hybrid_clock.cc:648] HybridClock initialized: now 1790388628087728 us; error 30 us; skew 500 ppm
I20260926 02:10:28.088138 1761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:28 dist-test-slave-03rn krb5kdc[444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388628, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:28.091619 1761 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:10:28.093278 1761 webserver.cc:460] Webserver started at http://127.18.159.130:35661/ using document root <none> and password file <none>
I20260926 02:10:28.094048 1761 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:28.094202 1761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:28.100034 1761 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260926 02:10:28.102942 1776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:28.104544 1761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:10:28.104744 1761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "9af40ba88bdc4287a6f472bfa69fc076"
format_stamp: "Formatted at 2026-09-26 02:10:22 on dist-test-slave-03rn"
server_key: "379a341e52e62de094db00a685fd168b"
server_key_iv: "b0cf1595c8b5d2a761791633d917429e"
server_key_version: "encryptionkey@0"
I20260926 02:10:28.105813 1761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:28.145028 1761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:28.149866 1761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:28.150167 1761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:28.151284 1761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:10:28.152763 1761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:10:28.152908 1761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:28.153039 1761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:10:28.153128 1761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:28.188077 1761 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:38413
I20260926 02:10:28.188134 1888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:38413 every 8 connection(s)
I20260926 02:10:28.189706 1761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:10:28.191545 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 1761
I20260926 02:10:28.192139 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:35439
--local_ip_for_outbound_sockets=127.18.159.131
--tserver_master_addrs=127.18.159.190:32955
--webserver_port=39045
--webserver_interface=127.18.159.131
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:10:28.198820 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:28.193047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.130:36887)
0926 02:10:28.197273 (+ 4226us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:28.197576 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3866,"thread_start_us":3623,"threads_started":1}
W20260926 02:10:28.199745 1889 heartbeater.cc:644] Failed to heartbeat to 127.18.159.190:32955 (0 consecutive failures): Network error: Failed to ping master at 127.18.159.190:32955: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
I20260926 02:10:28.200775 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:28.200421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.130:34555)
0926 02:10:28.200589 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:28.200662 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 02:10:28.201654 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:28.201310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.130:42623)
0926 02:10:28.201457 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:28.201544 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 02:10:28.201896 1889 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:10:28.542892 1892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:28.543258 1892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:28.543434 1892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:28.550871 1892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:28.551069 1892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:28.551268 1892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:10:28.561192 1892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:35439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=39045
--tserver_master_addrs=127.18.159.190:32955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:28.562582 1892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:28.563906 1892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:28.577445 1898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:28.577466 1897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:28.577762 1900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:28.578555 1892 server_base.cc:1034] running on GCE node
I20260926 02:10:28.579186 1892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:28.580226 1892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:28.581511 1892 hybrid_clock.cc:648] HybridClock initialized: now 1790388628581446 us; error 92 us; skew 500 ppm
I20260926 02:10:28.581887 1892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:28 dist-test-slave-03rn krb5kdc[444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388628, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:28.585232 1892 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:10:28.586833 1892 webserver.cc:460] Webserver started at http://127.18.159.131:39045/ using document root <none> and password file <none>
I20260926 02:10:28.587567 1892 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:28.587721 1892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:28.593508 1892 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:10:28.596680 1907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:28.598507 1892 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260926 02:10:28.598724 1892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "2601425df57849a6bbc056dcbf1ea73d"
format_stamp: "Formatted at 2026-09-26 02:10:22 on dist-test-slave-03rn"
server_key: "49454775f30aca5cf742b98b19d53ef4"
server_key_iv: "3115ddd3aef163aba5bba9d16edeb9cb"
server_key_version: "encryptionkey@0"
I20260926 02:10:28.599821 1892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:28.621200 1892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:28.626016 1892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:28.626324 1892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:28.627524 1892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:10:28.629001 1892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:10:28.629133 1892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:28.629271 1892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:10:28.629359 1892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:28.665515 1892 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:35439
I20260926 02:10:28.665565 2019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:35439 every 8 connection(s)
I20260926 02:10:28.667136 1892 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:10:28.667783 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 1892
I20260926 02:10:28.672611 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:28.670529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.131:55561)
0926 02:10:28.671321 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:28.671559 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":333,"threads_started":1}
W20260926 02:10:28.673496 2020 heartbeater.cc:644] Failed to heartbeat to 127.18.159.190:32955 (0 consecutive failures): Network error: Failed to ping master at 127.18.159.190:32955: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
I20260926 02:10:28.674562 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:28.674118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.131:39083)
0926 02:10:28.674303 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:28.674380 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 02:10:28.675589 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:28.675235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.131:35147)
0926 02:10:28.675366 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:28.675470 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 02:10:28.675904 2020 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 02:10:28.711227 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:28.710330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.129:54517)
0926 02:10:28.710908 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:28.710997 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":325,"threads_started":1}
I20260926 02:10:29.203807 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:29.202617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.130:44741)
0926 02:10:29.203365 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:29.203521 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":342,"threads_started":1}
I20260926 02:10:29.677615 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:29.676671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.131:33145)
0926 02:10:29.677231 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:29.677368 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":308,"threads_started":1}
I20260926 02:10:29.713231 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:29.712409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.129:52215)
0926 02:10:29.712882 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:29.713017 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":247,"threads_started":1}
I20260926 02:10:30.205634 2028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:30.204840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.130:52381)
0926 02:10:30.205315 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:30.205406 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":223,"threads_started":1}
I20260926 02:10:30.679625 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:30.678808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.131:41277)
0926 02:10:30.679328 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:30.679451 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":270,"threads_started":1}
I20260926 02:10:30.715227 2030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:30.714431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.129:52635)
0926 02:10:30.714954 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:30.715058 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":254,"threads_started":1}
I20260926 02:10:31.207515 2031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:31.206633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.130:51891)
0926 02:10:31.207202 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:31.207318 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":268,"threads_started":1}
I20260926 02:10:31.681772 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:31.680876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.131:46659)
0926 02:10:31.681385 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:31.681504 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":277,"threads_started":1}
I20260926 02:10:31.717206 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:31.716419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.129:48263)
0926 02:10:31.716930 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:31.717021 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":263,"threads_started":1}
I20260926 02:10:32.209570 2034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:32.208662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.130:55113)
0926 02:10:32.209238 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:32.209350 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":330,"threads_started":1}
I20260926 02:10:32.683624 2035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:32.682824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.131:50143)
0926 02:10:32.683316 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:32.683409 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":252,"threads_started":1}
I20260926 02:10:32.719058 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:32.718264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.129:40237)
0926 02:10:32.718776 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:32.718870 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":263,"threads_started":1}
I20260926 02:10:33.211287 2037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:33.210436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.130:50367)
0926 02:10:33.210935 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:33.211109 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":255,"threads_started":1}
I20260926 02:10:33.685567 2038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:33.684770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.131:43831)
0926 02:10:33.685285 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:33.685384 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":271,"threads_started":1}
I20260926 02:10:33.721319 2039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:33.720376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.129:54111)
0926 02:10:33.720934 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:33.721055 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":284,"threads_started":1}
I20260926 02:10:34.213028 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:34.212280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.130:50519)
0926 02:10:34.212764 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:34.212859 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":251,"threads_started":1}
I20260926 02:10:34.687347 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:34.686620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.131:35843)
0926 02:10:34.687089 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:34.687185 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":235,"threads_started":1}
I20260926 02:10:34.723268 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:34.722399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.129:41195)
0926 02:10:34.722967 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:34.723055 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":267,"threads_started":1}
I20260926 02:10:35.214862 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:35.214020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.130:48489)
0926 02:10:35.214564 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:35.214654 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":215,"threads_started":1}
I20260926 02:10:35.689337 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:35.688508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.131:56249)
0926 02:10:35.689015 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:35.689122 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":224,"threads_started":1}
I20260926 02:10:35.725584 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:35.724663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.129:34357)
0926 02:10:35.725225 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:35.725321 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":343,"threads_started":1}
I20260926 02:10:36.216754 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:36.215892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.130:56025)
0926 02:10:36.216493 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:36.216577 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":331,"threads_started":1}
I20260926 02:10:36.691219 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:36.690409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.131:60527)
0926 02:10:36.690959 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:36.691049 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":318,"threads_started":1}
I20260926 02:10:36.727397 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:36.726554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.129:37639)
0926 02:10:36.727147 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:36.727243 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":339,"threads_started":1}
I20260926 02:10:37.218405 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:37.217659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.130:39367)
0926 02:10:37.218139 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:37.218233 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":223,"threads_started":1}
I20260926 02:10:37.693235 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:37.692424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.131:47307)
0926 02:10:37.692971 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:37.693058 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":236,"threads_started":1}
I20260926 02:10:37.729574 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:37.728757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.129:41191)
0926 02:10:37.729277 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:37.729376 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":256,"threads_started":1}
I20260926 02:10:38.220461 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:38.219522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.130:53909)
0926 02:10:38.220029 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:38.220121 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":286,"threads_started":1}
I20260926 02:10:38.695128 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:38.694283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.131:48065)
0926 02:10:38.694852 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:38.694946 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":327,"threads_started":1}
I20260926 02:10:38.731503 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:38.730739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.129:37557)
0926 02:10:38.731229 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:38.731320 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":261,"threads_started":1}
I20260926 02:10:39.222394 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:39.221570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.130:59489)
0926 02:10:39.222095 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:39.222201 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":248,"threads_started":1}
I20260926 02:10:39.696986 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:39.696255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.131:43099)
0926 02:10:39.696708 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:39.696806 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":216,"threads_started":1}
I20260926 02:10:39.733395 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:39.732634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.129:37029)
0926 02:10:39.733128 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:39.733216 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":257,"threads_started":1}
I20260926 02:10:40.224548 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:40.223678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.130:57077)
0926 02:10:40.224290 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:40.224387 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":281,"threads_started":1}
I20260926 02:10:40.698890 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:40.698104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.131:42961)
0926 02:10:40.698634 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:40.698728 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":250,"threads_started":1}
I20260926 02:10:40.735622 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:40.734575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.129:54261)
0926 02:10:40.735244 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:40.735420 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":348,"threads_started":1}
I20260926 02:10:41.226508 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:41.225587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.130:60617)
0926 02:10:41.226167 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:41.226266 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":322,"threads_started":1}
I20260926 02:10:41.700920 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:41.700011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.131:40057)
0926 02:10:41.700584 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:41.700692 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":296,"threads_started":1}
I20260926 02:10:41.737437 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:41.736685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.129:59911)
0926 02:10:41.737154 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:41.737244 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":232,"threads_started":1}
I20260926 02:10:42.228418 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:42.227578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.130:57797)
0926 02:10:42.228077 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:42.228235 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":269,"threads_started":1}
I20260926 02:10:42.702742 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:42.701942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.131:48767)
0926 02:10:42.702470 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:42.702558 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":297,"threads_started":1}
I20260926 02:10:42.739696 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:42.738835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.129:41187)
0926 02:10:42.739350 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:42.739520 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":259,"threads_started":1}
I20260926 02:10:43.230631 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:43.229685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.130:33551)
0926 02:10:43.230272 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:43.230378 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":344,"threads_started":1}
I20260926 02:10:43.676182 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 1595
I20260926 02:10:43.703171 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:43.677442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48984 (local address 127.18.159.129:40959)
0926 02:10:43.678193 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:43.678209 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:43.678251 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:10:43.680641 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:43.680673 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:43.681241 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:43.681643 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:43.683229 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:43.684603 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:43.685947 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:43.686683 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:43.686897 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:43.695797 (+ 8900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:43.695808 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:43.695826 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:43.695881 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:43.699148 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:43.700051 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:43.700059 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:43.700067 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:43.700368 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:43.700827 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:43.700832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:43.700835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:43.701487 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:43.701824 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:43.702315 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 02:10:43.702802 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":420,"threads_started":1}
I20260926 02:10:43.704908 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:43.703966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:32955 (local address 127.18.159.131:47291)
0926 02:10:43.704569 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:43.704667 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":386,"threads_started":1}
I20260926 02:10:43.705259 1753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:43.793812 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 1595
I20260926 02:10:43.812182 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 1761
I20260926 02:10:43.837477 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:43.813498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.18.159.130:38413)
0926 02:10:43.814239 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:43.814268 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:43.814319 (+ 51us) server_negotiation.cc:407] Connection header received
0926 02:10:43.817224 (+ 2905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:43.817241 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:43.817957 (+ 716us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:43.818450 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:43.819959 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:43.822302 (+ 2343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:43.823781 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:43.825137 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:43.825363 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:43.828338 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:43.828352 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:43.828379 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:43.828482 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:43.833028 (+ 4546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:43.833851 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:43.833864 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:43.833877 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:43.834150 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:43.834562 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:43.834570 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:43.834574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:43.835437 (+ 863us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:43.835786 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:43.836355 (+ 569us) server_negotiation.cc:299] Negotiation successful
0926 02:10:43.836927 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":317,"threads_started":1}
I20260926 02:10:43.839154 1884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:43.947142 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 1761
I20260926 02:10:43.964697 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 1892
I20260926 02:10:43.985278 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:43.966012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59612 (local address 127.18.159.131:35439)
0926 02:10:43.966760 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:43.966774 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:43.966817 (+ 43us) server_negotiation.cc:407] Connection header received
0926 02:10:43.968568 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:43.968581 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:43.969065 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:43.969395 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:43.971073 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:43.972514 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:43.973936 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:43.974712 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:43.974875 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:43.977854 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:43.977862 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:43.977881 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:43.977931 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:43.981475 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:43.982386 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:43.982393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:43.982401 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:43.982635 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:43.983170 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:43.983174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:43.983176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:43.983791 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:43.984055 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:43.984584 (+ 529us) server_negotiation.cc:299] Negotiation successful
0926 02:10:43.984963 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":398,"threads_started":1}
I20260926 02:10:43.986747 2015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:44.072394 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 1892
2026-09-26T02:10:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25053 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:10:44 dist-test-slave-03rn krb5kdc[2078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:10:44 dist-test-slave-03rn krb5kdc[2078](info): set up 2 sockets
Sep 26 02:10:44 dist-test-slave-03rn krb5kdc[2078](info): commencing operation
krb5kdc: starting...
W20260926 02:10:45.181214 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:10:45 dist-test-slave-03rn krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:10:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:10:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:10:45.436244 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:44923
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:44923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:10:45.804055 2091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:45.804489 2091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:45.804589 2091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:45.812749 2091 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 02:10:45.812878 2091 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 02:10:45.812924 2091 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 02:10:45.812979 2091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:10:45.813037 2091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:45.813100 2091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:45.813148 2091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:10:45.813205 2091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:10:45.825028 2091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:44923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:44923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:45.826359 2091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:45.827693 2091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:45.840790 2097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:45.841074 2096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:45.840770 2099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:45.842386 2091 server_base.cc:1034] running on GCE node
I20260926 02:10:45.843255 2091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:45.844956 2091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:45.846244 2091 hybrid_clock.cc:648] HybridClock initialized: now 1790388645846163 us; error 67 us; skew 500 ppm
I20260926 02:10:45.846634 2091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:45 dist-test-slave-03rn krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388645, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:45.850144 2091 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:10:45.851972 2091 webserver.cc:460] Webserver started at http://127.18.159.190:34987/ using document root <none> and password file <none>
I20260926 02:10:45.852856 2091 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:45.853004 2091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:45.853351 2091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:10:45.856763 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "2f42205c49a844fc838a25f56c408269"
format_stamp: "Formatted at 2026-09-26 02:10:45 on dist-test-slave-03rn"
server_key: "3066a25d6c037c7866a55bdb4fd9c090"
server_key_iv: "1135d650f88efde7a9dd5526ef29b1b6"
server_key_version: "encryptionkey@0"
I20260926 02:10:45.857679 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "2f42205c49a844fc838a25f56c408269"
format_stamp: "Formatted at 2026-09-26 02:10:45 on dist-test-slave-03rn"
server_key: "3066a25d6c037c7866a55bdb4fd9c090"
server_key_iv: "1135d650f88efde7a9dd5526ef29b1b6"
server_key_version: "encryptionkey@0"
I20260926 02:10:45.863561 2091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 02:10:45.867899 2106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:45.869635 2091 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:10:45.869860 2091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "2f42205c49a844fc838a25f56c408269"
format_stamp: "Formatted at 2026-09-26 02:10:45 on dist-test-slave-03rn"
server_key: "3066a25d6c037c7866a55bdb4fd9c090"
server_key_iv: "1135d650f88efde7a9dd5526ef29b1b6"
server_key_version: "encryptionkey@0"
I20260926 02:10:45.870121 2091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:45.899533 2091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:45.904426 2091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:45.904769 2091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:45.925215 2091 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:44923
I20260926 02:10:45.925232 2157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:44923 every 8 connection(s)
I20260926 02:10:45.926908 2091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:10:45.931042 2158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:10:45.932996 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 2091
I20260926 02:10:45.933246 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:10:45.934078 19070 external_mini_cluster.cc:1428] Setting key 1a4c8877462956524c8f71f165f3eaba
I20260926 02:10:45.947412 2158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269: Bootstrap starting.
Sep 26 02:10:45 dist-test-slave-03rn krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:10:45.952224 2158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269: Neither blocks nor log segments found. Creating new log.
I20260926 02:10:45.953437 2158 log.cc:824] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269: Log is configured to *not* fsync() on all Append() calls
I20260926 02:10:45.957221 2158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269: No bootstrap required, opened a new log
I20260926 02:10:45.964879 2161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:45.936566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.18.159.190:44923)
0926 02:10:45.937510 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:45.937523 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:45.937573 (+ 50us) server_negotiation.cc:407] Connection header received
0926 02:10:45.939336 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:45.939369 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:45.939871 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:45.940333 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:45.942418 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:45.943900 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:45.946071 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:45.946895 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:45.947067 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:45.952334 (+ 5267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:45.952343 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:45.952385 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:45.952447 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:45.956052 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:45.957191 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:45.957201 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:45.957221 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:45.957523 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:45.958647 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:45.958672 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:45.958684 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:45.959535 (+ 851us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:45.959898 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:45.961891 (+ 1993us) server_negotiation.cc:299] Negotiation successful
0926 02:10:45.962551 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":445,"threads_started":1}
I20260926 02:10:45.971496 2158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f42205c49a844fc838a25f56c408269" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44923 } }
I20260926 02:10:45.971935 2158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:10:45.972054 2158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f42205c49a844fc838a25f56c408269, State: Initialized, Role: FOLLOWER
I20260926 02:10:45.972841 2158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f42205c49a844fc838a25f56c408269" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44923 } }
I20260926 02:10:45.973276 2158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:10:45.973470 2158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:10:45.973690 2158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:10:45.976620 2158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f42205c49a844fc838a25f56c408269" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44923 } }
I20260926 02:10:45.977293 2158 leader_election.cc:302] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f42205c49a844fc838a25f56c408269; no voters:
I20260926 02:10:45.978106 2158 leader_election.cc:288] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:10:45.978267 2163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:10:45.979923 2163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 1 LEADER]: Becoming Leader. State: Replica: 2f42205c49a844fc838a25f56c408269, State: Running, Role: LEADER
I20260926 02:10:45.980664 2158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:10:45.980805 2163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f42205c49a844fc838a25f56c408269" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44923 } }
I20260926 02:10:45.984138 2164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f42205c49a844fc838a25f56c408269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f42205c49a844fc838a25f56c408269" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44923 } } }
I20260926 02:10:45.984961 2164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [sys.catalog]: This master's current role is: LEADER
I20260926 02:10:45.984536 2165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f42205c49a844fc838a25f56c408269. Latest consensus state: current_term: 1 leader_uuid: "2f42205c49a844fc838a25f56c408269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f42205c49a844fc838a25f56c408269" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44923 } } }
I20260926 02:10:45.985746 2165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [sys.catalog]: This master's current role is: LEADER
I20260926 02:10:45.988456 2171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:10:45.994889 2171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:10:46.013753 2171 catalog_manager.cc:1347] Generated new cluster ID: 9f770d8049af48198830e14f9ddd7c03
I20260926 02:10:46.013908 2171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:10:46.038349 2171 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:10:46.040498 2171 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:10:46.059962 2171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269: Generated new TSK 0
I20260926 02:10:46.061192 2171 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.159.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
I20260926 02:10:46.128439 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:44923
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:10:46.498162 2184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:46.498471 2184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:46.498569 2184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:46.506301 2184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:46.506449 2184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:46.506568 2184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:10:46.517109 2184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:44923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:46.518380 2184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:46.519742 2184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:46.534070 2189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:46.533995 2190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:46.534106 2192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:46.535377 2184 server_base.cc:1034] running on GCE node
I20260926 02:10:46.536118 2184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:46.537241 2184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:46.538528 2184 hybrid_clock.cc:648] HybridClock initialized: now 1790388646538434 us; error 40 us; skew 500 ppm
I20260926 02:10:46.538898 2184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:46 dist-test-slave-03rn krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388646, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:46.542560 2184 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:10:46.544390 2184 webserver.cc:460] Webserver started at http://127.18.159.129:34625/ using document root <none> and password file <none>
I20260926 02:10:46.545168 2184 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:46.545311 2184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:46.545660 2184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:10:46.548908 2184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "79231b61c8954168950f91e8d03dd1f3"
format_stamp: "Formatted at 2026-09-26 02:10:46 on dist-test-slave-03rn"
server_key: "155e736d0429762f64ba6fe220c129f4"
server_key_iv: "3faeabb531c35e2ba0c4db64188b873f"
server_key_version: "encryptionkey@0"
I20260926 02:10:46.549772 2184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "79231b61c8954168950f91e8d03dd1f3"
format_stamp: "Formatted at 2026-09-26 02:10:46 on dist-test-slave-03rn"
server_key: "155e736d0429762f64ba6fe220c129f4"
server_key_iv: "3faeabb531c35e2ba0c4db64188b873f"
server_key_version: "encryptionkey@0"
I20260926 02:10:46.555773 2184 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260926 02:10:46.560091 2199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:46.561806 2184 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:10:46.562023 2184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "79231b61c8954168950f91e8d03dd1f3"
format_stamp: "Formatted at 2026-09-26 02:10:46 on dist-test-slave-03rn"
server_key: "155e736d0429762f64ba6fe220c129f4"
server_key_iv: "3faeabb531c35e2ba0c4db64188b873f"
server_key_version: "encryptionkey@0"
I20260926 02:10:46.562249 2184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:46.581208 2184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:46.586333 2184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:46.586700 2184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:46.588022 2184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:10:46.589970 2184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:10:46.590097 2184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:46.590281 2184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:10:46.590363 2184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:46.628960 2184 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:43893
I20260926 02:10:46.629004 2311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:43893 every 8 connection(s)
I20260926 02:10:46.630605 2184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:10:46.640756 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 2184
I20260926 02:10:46.641006 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:10:46.641805 19070 external_mini_cluster.cc:1428] Setting key 3f7459472e035c054e9045c80aeb03de
Sep 26 02:10:46 dist-test-slave-03rn krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388646, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:10:46.657678 2315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:46.635318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:60469 (local address 127.18.159.190:44923)
0926 02:10:46.635809 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:46.635817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:46.635985 (+ 168us) server_negotiation.cc:407] Connection header received
0926 02:10:46.638798 (+ 2813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:46.638803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:46.638900 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:46.639034 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:46.641460 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:46.642417 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:46.644730 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:46.645162 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:46.645336 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:46.650228 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:46.650239 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:46.650250 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:46.650310 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:46.653853 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:46.654906 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:46.654913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:46.654916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:46.655103 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:46.655661 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:46.655665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:46.655667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:46.655979 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:46.656153 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:46.657013 (+ 860us) server_negotiation.cc:299] Negotiation successful
0926 02:10:46.657281 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":1165,"threads_started":1}
I20260926 02:10:46.659081 2314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:46.634881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:60469)
0926 02:10:46.635768 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:46.635826 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:46.638424 (+ 2598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:46.639342 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:46.639359 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:46.640176 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:46.641196 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:46.641230 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:46.642638 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:46.642643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:46.644517 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:46.644534 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:46.645485 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:46.645495 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:46.645888 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:46.646783 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:46.646826 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:46.649842 (+ 3016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:46.654120 (+ 4278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:46.654140 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:46.654157 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:46.654722 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:46.655248 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:46.655255 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:46.655264 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:46.655508 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:46.656127 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:46.656136 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:46.656580 (+ 444us) client_negotiation.cc:769] Sending connection context
0926 02:10:46.656894 (+ 314us) client_negotiation.cc:240] Negotiation successful
0926 02:10:46.657543 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":519,"threads_started":1}
I20260926 02:10:46.661090 2312 heartbeater.cc:347] Connected to a master server at 127.18.159.190:44923
I20260926 02:10:46.661470 2312 heartbeater.cc:464] Registering TS with master...
I20260926 02:10:46.662298 2312 heartbeater.cc:511] Master 127.18.159.190:44923 requested a full tablet report, sending...
I20260926 02:10:46.665246 2123 ts_manager.cc:194] Registered new tserver with Master: 79231b61c8954168950f91e8d03dd1f3 (127.18.159.129:43893)
I20260926 02:10:46.667730 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:60469
WARNING: no policy specified for kudu/127.18.159.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
I20260926 02:10:46.702215 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:44923
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:10:47.058063 2318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:47.058373 2318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:47.058483 2318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:47.066278 2318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:47.066406 2318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:47.066573 2318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:10:47.077133 2318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:44923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:47.078449 2318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:47.079826 2318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:47.094334 2326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:47.094420 2324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:47.094372 2323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:47.097987 2318 server_base.cc:1034] running on GCE node
I20260926 02:10:47.098832 2318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:47.099974 2318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:47.101248 2318 hybrid_clock.cc:648] HybridClock initialized: now 1790388647101197 us; error 47 us; skew 500 ppm
I20260926 02:10:47.101606 2318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:47 dist-test-slave-03rn krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388647, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:47.105051 2318 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:10:47.106829 2318 webserver.cc:460] Webserver started at http://127.18.159.130:42565/ using document root <none> and password file <none>
I20260926 02:10:47.107611 2318 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:47.107764 2318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:47.108091 2318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:10:47.111281 2318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "4544ec3626644f498d99b4f9408d4eae"
format_stamp: "Formatted at 2026-09-26 02:10:47 on dist-test-slave-03rn"
server_key: "cf7875657dcbead532a0ee6fd496df76"
server_key_iv: "5e27c7ea240aee946dff8a6b1114ab60"
server_key_version: "encryptionkey@0"
I20260926 02:10:47.112133 2318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "4544ec3626644f498d99b4f9408d4eae"
format_stamp: "Formatted at 2026-09-26 02:10:47 on dist-test-slave-03rn"
server_key: "cf7875657dcbead532a0ee6fd496df76"
server_key_iv: "5e27c7ea240aee946dff8a6b1114ab60"
server_key_version: "encryptionkey@0"
I20260926 02:10:47.118006 2318 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:10:47.121999 2333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:47.123649 2318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:10:47.123867 2318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "4544ec3626644f498d99b4f9408d4eae"
format_stamp: "Formatted at 2026-09-26 02:10:47 on dist-test-slave-03rn"
server_key: "cf7875657dcbead532a0ee6fd496df76"
server_key_iv: "5e27c7ea240aee946dff8a6b1114ab60"
server_key_version: "encryptionkey@0"
I20260926 02:10:47.124116 2318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:47.154102 2318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:47.158897 2318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:47.159271 2318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:47.160459 2318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:10:47.161877 2318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:10:47.161988 2318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:47.162144 2318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:10:47.162225 2318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:47.197690 2318 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:34015
I20260926 02:10:47.197780 2445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:34015 every 8 connection(s)
I20260926 02:10:47.199046 2318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:10:47.208823 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 2318
I20260926 02:10:47.209038 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:10:47.209637 19070 external_mini_cluster.cc:1428] Setting key e5525f4f57e1c0ff188ac445febcf55c
Sep 26 02:10:47 dist-test-slave-03rn krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388647, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:10:47.223270 2448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:47.201862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:45487 (local address 127.18.159.190:44923)
0926 02:10:47.202363 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:47.202369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:47.203397 (+ 1028us) server_negotiation.cc:407] Connection header received
0926 02:10:47.205597 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:47.205603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:47.205703 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:47.205824 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:47.208324 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.209249 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.211500 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.212058 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.212262 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:47.216898 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:47.216903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:47.216907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:47.216945 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:47.219779 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:47.220763 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:47.220768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:47.220771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:47.220908 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:47.221420 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:47.221424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:47.221426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:47.221722 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:47.221883 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:47.222625 (+ 742us) server_negotiation.cc:299] Negotiation successful
0926 02:10:47.222868 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":246,"threads_started":1}
I20260926 02:10:47.224376 2449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:47.202480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.130:45487)
0926 02:10:47.203216 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:47.203254 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:47.205233 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:47.206216 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:47.206228 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:47.207084 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:47.208061 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:47.208090 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.209423 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.209428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:47.211297 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:47.211310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.212379 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.212387 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:47.212771 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:47.213541 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:47.213579 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:47.216585 (+ 3006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:47.219979 (+ 3394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:47.219989 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:47.220017 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:47.220596 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:47.221036 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:47.221041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:47.221045 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:47.221263 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:47.221858 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:47.221866 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:47.222246 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 02:10:47.222521 (+ 275us) client_negotiation.cc:240] Negotiation successful
0926 02:10:47.223117 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":284,"threads_started":1}
I20260926 02:10:47.226037 2446 heartbeater.cc:347] Connected to a master server at 127.18.159.190:44923
I20260926 02:10:47.226395 2446 heartbeater.cc:464] Registering TS with master...
I20260926 02:10:47.227416 2446 heartbeater.cc:511] Master 127.18.159.190:44923 requested a full tablet report, sending...
I20260926 02:10:47.229239 2123 ts_manager.cc:194] Registered new tserver with Master: 4544ec3626644f498d99b4f9408d4eae (127.18.159.130:34015)
I20260926 02:10:47.230131 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:45487
WARNING: no policy specified for kudu/127.18.159.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
I20260926 02:10:47.269596 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:44923
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:10:47.615636 2452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:10:47.615942 2452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:10:47.616039 2452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:10:47.623572 2452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:10:47.623711 2452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:10:47.623831 2452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:10:47.634701 2452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:44923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:10:47.635973 2452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:10:47.637395 2452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:10:47.652029 2458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:47.652120 2460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:10:47.652061 2457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:10:47.652845 2452 server_base.cc:1034] running on GCE node
I20260926 02:10:47.653589 2452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:10:47.654611 2452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:10:47.655800 2452 hybrid_clock.cc:648] HybridClock initialized: now 1790388647655749 us; error 44 us; skew 500 ppm
I20260926 02:10:47.656219 2452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:10:47 dist-test-slave-03rn krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388647, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:10:47.659466 2452 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:10:47.661065 2452 webserver.cc:460] Webserver started at http://127.18.159.131:39463/ using document root <none> and password file <none>
I20260926 02:10:47.661829 2452 fs_manager.cc:359] Metadata directory not provided
I20260926 02:10:47.661985 2452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:10:47.662313 2452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:10:47.665509 2452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "8ebf3a462272420e821e8b9e2f8798ae"
format_stamp: "Formatted at 2026-09-26 02:10:47 on dist-test-slave-03rn"
server_key: "978cacf6acdcc7c73fae1fcb68955816"
server_key_iv: "8ed615e48ee58f20ac8b64f712bee89e"
server_key_version: "encryptionkey@0"
I20260926 02:10:47.666491 2452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "8ebf3a462272420e821e8b9e2f8798ae"
format_stamp: "Formatted at 2026-09-26 02:10:47 on dist-test-slave-03rn"
server_key: "978cacf6acdcc7c73fae1fcb68955816"
server_key_iv: "8ed615e48ee58f20ac8b64f712bee89e"
server_key_version: "encryptionkey@0"
I20260926 02:10:47.671832 2312 heartbeater.cc:503] Master 127.18.159.190:44923 was elected leader, sending a full tablet report...
I20260926 02:10:47.672578 2452 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 02:10:47.676743 2467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:47.678377 2452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:10:47.678601 2452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "8ebf3a462272420e821e8b9e2f8798ae"
format_stamp: "Formatted at 2026-09-26 02:10:47 on dist-test-slave-03rn"
server_key: "978cacf6acdcc7c73fae1fcb68955816"
server_key_iv: "8ed615e48ee58f20ac8b64f712bee89e"
server_key_version: "encryptionkey@0"
I20260926 02:10:47.678851 2452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:10:47.707199 2452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:10:47.711660 2452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:10:47.711982 2452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:10:47.713043 2452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:10:47.714533 2452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:10:47.714635 2452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:47.714788 2452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:10:47.714859 2452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:10:47.749711 2452 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:38029
I20260926 02:10:47.749799 2579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:38029 every 8 connection(s)
I20260926 02:10:47.751062 2452 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:10:47.755522 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 2452
I20260926 02:10:47.755729 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:10:47.756238 19070 external_mini_cluster.cc:1428] Setting key bda686dc86f6eded158435e142bf723c
Sep 26 02:10:47 dist-test-slave-03rn krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388647, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:10:47.773087 2582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:47.753862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:58069 (local address 127.18.159.190:44923)
0926 02:10:47.754344 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:47.754349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:47.755525 (+ 1176us) server_negotiation.cc:407] Connection header received
0926 02:10:47.758053 (+ 2528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:47.758058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:47.758169 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:47.758293 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:47.759954 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.760764 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.762397 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.762831 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.763011 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:47.766787 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:47.766793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:47.766796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:47.766835 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:47.769713 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:47.770529 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:47.770535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:47.770540 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:47.770674 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:47.771143 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:47.771149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:47.771152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:47.771478 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:47.771657 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:47.772454 (+ 797us) server_negotiation.cc:299] Negotiation successful
0926 02:10:47.772687 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":232,"threads_started":1}
I20260926 02:10:47.774150 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:47.754466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.131:58069)
0926 02:10:47.755255 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:47.755304 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:47.757772 (+ 2468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:47.758565 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:47.758575 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:47.759123 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:10:47.759761 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:47.759786 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.760932 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.760936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:47.762239 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:47.762252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.763067 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.763074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:47.763339 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:47.764088 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:10:47.764113 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:10:47.766528 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:10:47.769881 (+ 3353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:47.769891 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:47.769912 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:47.770385 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:47.770794 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:10:47.770799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:10:47.770803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:10:47.771021 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:10:47.771626 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:10:47.771637 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:10:47.772017 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 02:10:47.772358 (+ 341us) client_negotiation.cc:240] Negotiation successful
0926 02:10:47.772944 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":312,"threads_started":1}
I20260926 02:10:47.775681 2580 heartbeater.cc:347] Connected to a master server at 127.18.159.190:44923
I20260926 02:10:47.775962 2580 heartbeater.cc:464] Registering TS with master...
I20260926 02:10:47.776753 2580 heartbeater.cc:511] Master 127.18.159.190:44923 requested a full tablet report, sending...
I20260926 02:10:47.778256 2123 ts_manager.cc:194] Registered new tserver with Master: 8ebf3a462272420e821e8b9e2f8798ae (127.18.159.131:38029)
I20260926 02:10:47.779101 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:58069
I20260926 02:10:47.784053 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:10:47.802064 2582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:47.788751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44020 (local address 127.18.159.190:44923)
0926 02:10:47.788924 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:47.788928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:47.789398 (+ 470us) server_negotiation.cc:407] Connection header received
0926 02:10:47.789619 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:47.789628 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:47.789725 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:47.789864 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:47.791525 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.792370 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.793711 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.794118 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.794216 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:47.796685 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:47.796690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:47.796693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:47.796731 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:47.799454 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:47.800249 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:47.800254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:47.800256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:47.800400 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:47.800764 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:47.800767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:47.800769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:47.801127 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:47.801321 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:47.801636 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 02:10:47.801827 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:10:47.806376 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:10:47.813448 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:10:47.854714 2594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:47.826761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57822 (local address 127.18.159.129:43893)
0926 02:10:47.842057 (+ 15296us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:47.842072 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:47.842114 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:10:47.842256 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:47.842270 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:47.842654 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:47.842928 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:10:47.846661 (+ 3733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.848089 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.851297 (+ 3208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.852910 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.853133 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:47.853329 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:47.854149 (+ 820us) server_negotiation.cc:299] Negotiation successful
0926 02:10:47.854330 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15049,"thread_start_us":18810,"threads_started":1}
I20260926 02:10:47.854833 2596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:47.827381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37084 (local address 127.18.159.131:38029)
0926 02:10:47.839920 (+ 12539us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:47.839934 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:47.845737 (+ 5803us) server_negotiation.cc:407] Connection header received
0926 02:10:47.845852 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:47.845863 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:47.846162 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:47.846373 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:10:47.847386 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.849051 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.851626 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.852909 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.853123 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:47.853318 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:47.854280 (+ 962us) server_negotiation.cc:299] Negotiation successful
0926 02:10:47.854463 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12263,"thread_start_us":12574,"threads_started":1}
I20260926 02:10:47.854790 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:47.826682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:43893 (local address 127.0.0.1:57822)
0926 02:10:47.839541 (+ 12859us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:47.839633 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:47.839951 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:47.843747 (+ 3796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:47.843760 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:47.843811 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:10:47.844422 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:47.844442 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.848274 (+ 3832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.848280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:47.851084 (+ 2804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:47.851103 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.853570 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.853577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:47.853867 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:47.853956 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 02:10:47.854096 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 02:10:47.854328 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12225,"mutex_wait_us":73,"thread_start_us":13506,"threads_started":1}
I20260926 02:10:47.855422 2595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:47.841418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:34015 (local address 127.0.0.1:46216)
0926 02:10:47.846210 (+ 4792us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:47.846229 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:47.846345 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:47.847935 (+ 1590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:47.847940 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:47.847966 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:10:47.848466 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:47.848477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.850318 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.850324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:47.852363 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:47.852379 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.854499 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.854515 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:47.854751 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:47.854790 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 02:10:47.854879 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 02:10:47.855017 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4649,"mutex_wait_us":21,"thread_start_us":4067,"threads_started":1}
I20260926 02:10:47.856057 2598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:47.841751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46216 (local address 127.18.159.130:34015)
0926 02:10:47.847244 (+ 5493us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:47.847251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:47.847307 (+ 56us) server_negotiation.cc:407] Connection header received
0926 02:10:47.847404 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:47.847411 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:47.847695 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:47.847927 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:10:47.848665 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.850154 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.852549 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.854019 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.854183 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:47.854341 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:47.855696 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0926 02:10:47.855839 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5325,"thread_start_us":5205,"threads_started":1}
I20260926 02:10:47.857192 2597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:47.827032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:38029 (local address 127.0.0.1:37084)
0926 02:10:47.845265 (+ 18233us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:47.845287 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:47.845422 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:47.846713 (+ 1291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:47.846720 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:47.846756 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:10:47.847151 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:47.847164 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.849236 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.849243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:47.851462 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:47.851473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.853499 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.853507 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:47.853875 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:47.853967 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 02:10:47.854103 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 02:10:47.856828 (+ 2725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18088,"mutex_wait_us":2634,"thread_start_us":18881,"threads_started":1}
I20260926 02:10:47.859491 2247 tablet_service.cc:1467] Processing CreateTablet for tablet b8eb9042ee9e4b39ae56c6c4c87ece9d (DEFAULT_TABLE table=test-table [id=40f68ada538546b688104ac0dbb3fa1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:10:47.859493 2381 tablet_service.cc:1467] Processing CreateTablet for tablet b8eb9042ee9e4b39ae56c6c4c87ece9d (DEFAULT_TABLE table=test-table [id=40f68ada538546b688104ac0dbb3fa1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:10:47.860806 2381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8eb9042ee9e4b39ae56c6c4c87ece9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:10:47.860931 2247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8eb9042ee9e4b39ae56c6c4c87ece9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:10:47.863168 2515 tablet_service.cc:1467] Processing CreateTablet for tablet b8eb9042ee9e4b39ae56c6c4c87ece9d (DEFAULT_TABLE table=test-table [id=40f68ada538546b688104ac0dbb3fa1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:10:47.864827 2515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8eb9042ee9e4b39ae56c6c4c87ece9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:10:47.873827 2599 tablet_bootstrap.cc:492] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae: Bootstrap starting.
I20260926 02:10:47.876315 2600 tablet_bootstrap.cc:492] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3: Bootstrap starting.
I20260926 02:10:47.879501 2599 tablet_bootstrap.cc:654] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae: Neither blocks nor log segments found. Creating new log.
I20260926 02:10:47.880177 2600 tablet_bootstrap.cc:654] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3: Neither blocks nor log segments found. Creating new log.
I20260926 02:10:47.880319 2601 tablet_bootstrap.cc:492] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae: Bootstrap starting.
I20260926 02:10:47.881299 2599 log.cc:824] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae: Log is configured to *not* fsync() on all Append() calls
I20260926 02:10:47.881440 2600 log.cc:824] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3: Log is configured to *not* fsync() on all Append() calls
I20260926 02:10:47.884326 2601 tablet_bootstrap.cc:654] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae: Neither blocks nor log segments found. Creating new log.
I20260926 02:10:47.884884 2599 tablet_bootstrap.cc:492] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae: No bootstrap required, opened a new log
I20260926 02:10:47.885022 2600 tablet_bootstrap.cc:492] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3: No bootstrap required, opened a new log
I20260926 02:10:47.885212 2599 ts_tablet_manager.cc:1397] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260926 02:10:47.885305 2600 ts_tablet_manager.cc:1397] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260926 02:10:47.885740 2601 log.cc:824] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae: Log is configured to *not* fsync() on all Append() calls
I20260926 02:10:47.894335 2599 raft_consensus.cc:348] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } }
I20260926 02:10:47.894785 2599 raft_consensus.cc:374] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:10:47.894883 2599 raft_consensus.cc:729] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4544ec3626644f498d99b4f9408d4eae, State: Initialized, Role: FOLLOWER
I20260926 02:10:47.895604 2601 tablet_bootstrap.cc:492] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae: No bootstrap required, opened a new log
I20260926 02:10:47.895577 2599 consensus_queue.cc:260] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } }
I20260926 02:10:47.895928 2601 ts_tablet_manager.cc:1397] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.010s
I20260926 02:10:47.896824 2446 heartbeater.cc:503] Master 127.18.159.190:44923 was elected leader, sending a full tablet report...
I20260926 02:10:47.897701 2600 raft_consensus.cc:348] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } }
I20260926 02:10:47.898190 2600 raft_consensus.cc:374] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:10:47.898289 2600 raft_consensus.cc:729] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79231b61c8954168950f91e8d03dd1f3, State: Initialized, Role: FOLLOWER
I20260926 02:10:47.898950 2600 consensus_queue.cc:260] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } }
I20260926 02:10:47.899600 2599 ts_tablet_manager.cc:1428] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260926 02:10:47.901319 2600 ts_tablet_manager.cc:1428] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260926 02:10:47.906504 2601 raft_consensus.cc:348] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } }
I20260926 02:10:47.906914 2601 raft_consensus.cc:374] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:10:47.907024 2601 raft_consensus.cc:729] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ebf3a462272420e821e8b9e2f8798ae, State: Initialized, Role: FOLLOWER
I20260926 02:10:47.907630 2601 consensus_queue.cc:260] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } }
I20260926 02:10:47.908738 2580 heartbeater.cc:503] Master 127.18.159.190:44923 was elected leader, sending a full tablet report...
I20260926 02:10:47.909305 2601 ts_tablet_manager.cc:1428] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
W20260926 02:10:47.951222 2447 tablet.cc:2369] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:10:47.989620 2606 raft_consensus.cc:482] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:10:47.989915 2606 raft_consensus.cc:504] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } }
I20260926 02:10:47.991753 2606 leader_election.cc:288] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4544ec3626644f498d99b4f9408d4eae (127.18.159.130:34015), 8ebf3a462272420e821e8b9e2f8798ae (127.18.159.131:38029)
I20260926 02:10:47.999210 2598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:47.992340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:43511 (local address 127.18.159.130:34015)
0926 02:10:47.992711 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:47.992720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:47.993465 (+ 745us) server_negotiation.cc:407] Connection header received
0926 02:10:47.993632 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:47.993637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:47.993717 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:47.993810 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:10:47.994505 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.995255 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.997344 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.998157 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.998336 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:47.998432 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:47.998775 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 02:10:47.998913 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260926 02:10:47.999217 2596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:47.992255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:34927 (local address 127.18.159.131:38029)
0926 02:10:47.992487 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:47.992494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:47.993652 (+ 1158us) server_negotiation.cc:407] Connection header received
0926 02:10:47.993821 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:47.993826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:47.993906 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:47.994011 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:10:47.994686 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.995449 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.997379 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.998163 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.998367 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:47.998445 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:47.998827 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 02:10:47.998958 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:10:47.999361 2609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:47.992250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:38029 (local address 127.18.159.129:34927)
0926 02:10:47.993478 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:47.993498 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:47.993668 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:47.994052 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:47.994055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:47.994075 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:10:47.994485 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:47.994496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.995607 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.995611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:47.997206 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:47.997218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.998473 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.998486 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:47.998638 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:47.998660 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 02:10:47.998777 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 02:10:47.998907 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1081,"mutex_wait_us":60,"thread_start_us":808,"threads_started":1}
I20260926 02:10:47.999632 2608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:47.991992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:34015 (local address 127.18.159.129:43511)
0926 02:10:47.993335 (+ 1343us) negotiation.cc:107] Waiting for socket to connect
0926 02:10:47.993352 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:10:47.993490 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:10:47.993844 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:10:47.993848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:10:47.993872 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:10:47.994344 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:47.994354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.995391 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.995397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:47.997176 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:10:47.997189 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:10:47.998419 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:10:47.998427 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:10:47.998601 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:47.998628 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 02:10:47.998707 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 02:10:47.998865 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1222,"mutex_wait_us":90,"thread_start_us":960,"threads_started":1}
I20260926 02:10:48.000495 2535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8eb9042ee9e4b39ae56c6c4c87ece9d" candidate_uuid: "79231b61c8954168950f91e8d03dd1f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebf3a462272420e821e8b9e2f8798ae" is_pre_election: true
I20260926 02:10:48.000765 2401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8eb9042ee9e4b39ae56c6c4c87ece9d" candidate_uuid: "79231b61c8954168950f91e8d03dd1f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4544ec3626644f498d99b4f9408d4eae" is_pre_election: true
I20260926 02:10:48.001040 2535 raft_consensus.cc:2454] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79231b61c8954168950f91e8d03dd1f3 in term 0.
I20260926 02:10:48.001340 2401 raft_consensus.cc:2454] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79231b61c8954168950f91e8d03dd1f3 in term 0.
I20260926 02:10:48.001852 2200 leader_election.cc:302] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79231b61c8954168950f91e8d03dd1f3, 8ebf3a462272420e821e8b9e2f8798ae; no voters:
I20260926 02:10:48.002332 2606 raft_consensus.cc:2781] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:10:48.002588 2606 raft_consensus.cc:482] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:10:48.002719 2606 raft_consensus.cc:3037] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 [term 0 FOLLOWER]: Advancing to term 1
W20260926 02:10:48.002967 2581 tablet.cc:2369] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:10:48.005102 2606 raft_consensus.cc:504] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } }
I20260926 02:10:48.005838 2606 leader_election.cc:288] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 1 election: Requested vote from peers 4544ec3626644f498d99b4f9408d4eae (127.18.159.130:34015), 8ebf3a462272420e821e8b9e2f8798ae (127.18.159.131:38029)
I20260926 02:10:48.006356 2535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8eb9042ee9e4b39ae56c6c4c87ece9d" candidate_uuid: "79231b61c8954168950f91e8d03dd1f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebf3a462272420e821e8b9e2f8798ae"
I20260926 02:10:48.006348 2401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8eb9042ee9e4b39ae56c6c4c87ece9d" candidate_uuid: "79231b61c8954168950f91e8d03dd1f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4544ec3626644f498d99b4f9408d4eae"
I20260926 02:10:48.006569 2535 raft_consensus.cc:3037] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:10:48.006592 2401 raft_consensus.cc:3037] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:10:48.008911 2401 raft_consensus.cc:2454] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79231b61c8954168950f91e8d03dd1f3 in term 1.
I20260926 02:10:48.008983 2535 raft_consensus.cc:2454] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79231b61c8954168950f91e8d03dd1f3 in term 1.
I20260926 02:10:48.009444 2200 leader_election.cc:302] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79231b61c8954168950f91e8d03dd1f3, 8ebf3a462272420e821e8b9e2f8798ae; no voters:
I20260926 02:10:48.009769 2606 raft_consensus.cc:2781] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:10:48.010205 2606 raft_consensus.cc:686] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 [term 1 LEADER]: Becoming Leader. State: Replica: 79231b61c8954168950f91e8d03dd1f3, State: Running, Role: LEADER
I20260926 02:10:48.010694 2606 consensus_queue.cc:237] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } }
I20260926 02:10:48.015580 2122 catalog_manager.cc:5495] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79231b61c8954168950f91e8d03dd1f3 (127.18.159.129). New cstate: current_term: 1 leader_uuid: "79231b61c8954168950f91e8d03dd1f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } health_report { overall_health: UNKNOWN } } }
I20260926 02:10:48.051934 2594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:48.046710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57838 (local address 127.18.159.129:43893)
0926 02:10:48.046889 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:48.046894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:48.046913 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:10:48.047078 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:48.047082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:48.047183 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:48.047323 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:10:48.047674 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:48.048455 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:48.049883 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:48.050295 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:48.050386 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:48.050896 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:10:48.051410 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:10:48.051519 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:48.051652 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 02:10:48.051724 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 02:10:48.060796 2535 raft_consensus.cc:1261] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae [term 1 FOLLOWER]: Refusing update from remote peer 79231b61c8954168950f91e8d03dd1f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:10:48.060810 2401 raft_consensus.cc:1261] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae [term 1 FOLLOWER]: Refusing update from remote peer 79231b61c8954168950f91e8d03dd1f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:10:48.062040 2610 consensus_queue.cc:1035] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:10:48.062433 2606 consensus_queue.cc:1035] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:10:48.075019 2613 mvcc.cc:205] Tried to move back new op lower bound from 7333431902446063616 to 7333431902255960064. Current Snapshot: MvccSnapshot[applied={T|T < 7333431902446063616}]
I20260926 02:10:48.089542 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:10:48.089867 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:10:48.093989 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269: Sending DeleteTablet for 3 replicas of tablet b8eb9042ee9e4b39ae56c6c4c87ece9d
I20260926 02:10:48.095600 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 2184
I20260926 02:10:48.095548 2515 tablet_service.cc:1514] Processing DeleteTablet for tablet b8eb9042ee9e4b39ae56c6c4c87ece9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:10:48 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:37084
I20260926 02:10:48.095562 2381 tablet_service.cc:1514] Processing DeleteTablet for tablet b8eb9042ee9e4b39ae56c6c4c87ece9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:10:48 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:46216
I20260926 02:10:48.095821 2247 tablet_service.cc:1514] Processing DeleteTablet for tablet b8eb9042ee9e4b39ae56c6c4c87ece9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:10:48 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:57822
I20260926 02:10:48.106745 2625 tablet_replica.cc:330] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3: stopping tablet replica
I20260926 02:10:48.107659 2625 raft_consensus.cc:2229] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:10:48.108794 2625 raft_consensus.cc:2258] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:10:48.109894 2624 tablet_replica.cc:330] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae: stopping tablet replica
I20260926 02:10:48.110203 2623 tablet_replica.cc:330] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae: stopping tablet replica
I20260926 02:10:48.110638 2623 raft_consensus.cc:2229] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:10:48.111361 2623 raft_consensus.cc:2258] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:10:48.112085 2624 raft_consensus.cc:2229] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:10:48.112668 2624 raft_consensus.cc:2258] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:10:48.112851 2625 ts_tablet_manager.cc:1905] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:10:48.114516 2623 ts_tablet_manager.cc:1905] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:10:48.114638 2624 ts_tablet_manager.cc:1905] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:10:48.121435 2625 ts_tablet_manager.cc:1918] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:10:48.121680 2625 log.cc:1196] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/b8eb9042ee9e4b39ae56c6c4c87ece9d
I20260926 02:10:48.121752 2623 ts_tablet_manager.cc:1918] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:10:48.121927 2623 log.cc:1196] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/b8eb9042ee9e4b39ae56c6c4c87ece9d
I20260926 02:10:48.121954 2624 ts_tablet_manager.cc:1918] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:10:48.122154 2624 log.cc:1196] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/b8eb9042ee9e4b39ae56c6c4c87ece9d
I20260926 02:10:48.122262 2625 ts_tablet_manager.cc:1939] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 79231b61c8954168950f91e8d03dd1f3: Deleting consensus metadata
I20260926 02:10:48.122448 2623 ts_tablet_manager.cc:1939] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 4544ec3626644f498d99b4f9408d4eae: Deleting consensus metadata
I20260926 02:10:48.122696 2624 ts_tablet_manager.cc:1939] T b8eb9042ee9e4b39ae56c6c4c87ece9d P 8ebf3a462272420e821e8b9e2f8798ae: Deleting consensus metadata
I20260926 02:10:48.124636 2108 catalog_manager.cc:4841] TS 4544ec3626644f498d99b4f9408d4eae (127.18.159.130:34015): tablet b8eb9042ee9e4b39ae56c6c4c87ece9d (table test-table [id=40f68ada538546b688104ac0dbb3fa1f]) successfully deleted
I20260926 02:10:48.124752 2107 catalog_manager.cc:4841] TS 8ebf3a462272420e821e8b9e2f8798ae (127.18.159.131:38029): tablet b8eb9042ee9e4b39ae56c6c4c87ece9d (table test-table [id=40f68ada538546b688104ac0dbb3fa1f]) successfully deleted
I20260926 02:10:48.125551 2108 catalog_manager.cc:4841] TS 79231b61c8954168950f91e8d03dd1f3 (127.18.159.129:43893): tablet b8eb9042ee9e4b39ae56c6c4c87ece9d (table test-table [id=40f68ada538546b688104ac0dbb3fa1f]) successfully deleted
Sep 26 02:10:48 dist-test-slave-03rn krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:10:48.139878 2594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:48.096813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57848 (local address 127.18.159.129:43893)
0926 02:10:48.097057 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:48.097066 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:48.115346 (+ 18280us) server_negotiation.cc:407] Connection header received
0926 02:10:48.115600 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:48.115607 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:48.115707 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:48.115815 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:48.123032 (+ 7217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:48.124259 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:48.126612 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:48.127423 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:48.127568 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:48.132188 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:48.132206 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:48.132227 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:48.132291 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:48.136520 (+ 4229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:48.137370 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:48.137383 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:48.137388 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:48.137598 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:48.137990 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:48.137996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:48.138000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:48.138548 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:48.138866 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:48.139194 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 02:10:48.139473 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:10:48.140825 2307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:48.244844 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 2184
I20260926 02:10:48.263322 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 2318
Sep 26 02:10:48 dist-test-slave-03rn krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.130@KRBTEST.COM
I20260926 02:10:48.281281 2598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:48.264115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46220 (local address 127.18.159.130:34015)
0926 02:10:48.264355 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:48.264362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:48.264436 (+ 74us) server_negotiation.cc:407] Connection header received
0926 02:10:48.264701 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:48.264712 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:48.264779 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:48.264892 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:48.266221 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:48.267005 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:48.268488 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:48.268914 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:48.269079 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:48.273975 (+ 4896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:48.273992 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:48.274009 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:48.274059 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:48.277781 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:48.278757 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:48.278767 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:48.278769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:48.278925 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:48.279406 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:48.279411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:48.279413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:48.279835 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:48.280213 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:48.280635 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 02:10:48.280890 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260926 02:10:48.282095 2441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:48.378158 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 2318
I20260926 02:10:48.395900 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 2452
Sep 26 02:10:48 dist-test-slave-03rn krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.131@KRBTEST.COM
I20260926 02:10:48.412890 2596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:10:48.396758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37090 (local address 127.18.159.131:38029)
0926 02:10:48.396976 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 02:10:48.396982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:10:48.397028 (+ 46us) server_negotiation.cc:407] Connection header received
0926 02:10:48.397250 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:10:48.397255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:10:48.397317 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:10:48.397427 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:10:48.398813 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:48.399572 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:48.401045 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:10:48.401479 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:10:48.401629 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:10:48.406579 (+ 4950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:10:48.406596 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:10:48.406621 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:10:48.406671 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:10:48.409735 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:48.410541 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:48.410548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:48.410550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:48.410721 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:10:48.411177 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:10:48.411181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:10:48.411183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:10:48.411572 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:10:48.411819 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 02:10:48.412295 (+ 476us) server_negotiation.cc:299] Negotiation successful
0926 02:10:48.412481 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:10:48.413676 2575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:48.506734 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 2452
I20260926 02:10:48.523550 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 2091
I20260926 02:10:48.524856 2153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:10:48.638967 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 2091
I20260926 02:11:08.661803 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:44923
--webserver_interface=127.18.159.190
--webserver_port=34987
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:44923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:11:09.036369 2631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:11:09.036706 2631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:11:09.036875 2631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:11:09.044754 2631 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 02:11:09.044876 2631 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 02:11:09.044916 2631 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 02:11:09.045030 2631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:11:09.045107 2631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:11:09.045222 2631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:11:09.045302 2631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:11:09.045387 2631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:11:09.056248 2631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:44923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:44923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=34987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:11:09.057689 2631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:11:09.059135 2631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:11:09.084100 2638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:11:09.084225 2631 server_base.cc:1034] running on GCE node
W20260926 02:11:09.084371 2640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:09.084231 2637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:11:09.085623 2631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:11:09.087545 2631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:11:09.088809 2631 hybrid_clock.cc:648] HybridClock initialized: now 1790388669088725 us; error 72 us; skew 500 ppm
I20260926 02:11:09.089239 2631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:11:09 dist-test-slave-03rn krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388669, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:11:09.093305 2631 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:11:09.095311 2631 webserver.cc:460] Webserver started at http://127.18.159.190:34987/ using document root <none> and password file <none>
I20260926 02:11:09.096294 2631 fs_manager.cc:359] Metadata directory not provided
I20260926 02:11:09.096459 2631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:11:09.103361 2631 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 02:11:09.106925 2647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:09.108637 2631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:11:09.108879 2631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "2f42205c49a844fc838a25f56c408269"
format_stamp: "Formatted at 2026-09-26 02:10:45 on dist-test-slave-03rn"
server_key: "3066a25d6c037c7866a55bdb4fd9c090"
server_key_iv: "1135d650f88efde7a9dd5526ef29b1b6"
server_key_version: "encryptionkey@0"
I20260926 02:11:09.110071 2631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:11:09.127714 2631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:11:09.133210 2631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:11:09.133615 2631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:11:09.155484 2631 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:44923
I20260926 02:11:09.155507 2698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:44923 every 8 connection(s)
I20260926 02:11:09.157296 2631 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:11:09.160746 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 2631
I20260926 02:11:09.161806 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:43893
--local_ip_for_outbound_sockets=127.18.159.129
--tserver_master_addrs=127.18.159.190:44923
--webserver_port=34625
--webserver_interface=127.18.159.129
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:11:09.168121 2699 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:11:09.173167 2699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269: Bootstrap starting.
I20260926 02:11:09.195132 2699 log.cc:824] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269: Log is configured to *not* fsync() on all Append() calls
I20260926 02:11:09.208716 2699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:11:09.209345 2699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269: Bootstrap complete.
I20260926 02:11:09.219605 2699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f42205c49a844fc838a25f56c408269" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44923 } }
I20260926 02:11:09.220666 2699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f42205c49a844fc838a25f56c408269, State: Initialized, Role: FOLLOWER
I20260926 02:11:09.221431 2699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f42205c49a844fc838a25f56c408269" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44923 } }
I20260926 02:11:09.221755 2699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:11:09.221912 2699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:11:09.222103 2699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:11:09.226400 2699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f42205c49a844fc838a25f56c408269" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44923 } }
I20260926 02:11:09.227043 2699 leader_election.cc:302] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f42205c49a844fc838a25f56c408269; no voters:
I20260926 02:11:09.227716 2699 leader_election.cc:288] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 02:11:09.227870 2703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 02:11:09.229374 2703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 2 LEADER]: Becoming Leader. State: Replica: 2f42205c49a844fc838a25f56c408269, State: Running, Role: LEADER
I20260926 02:11:09.229988 2703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f42205c49a844fc838a25f56c408269" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44923 } }
I20260926 02:11:09.230329 2699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:11:09.232810 2704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2f42205c49a844fc838a25f56c408269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f42205c49a844fc838a25f56c408269" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44923 } } }
I20260926 02:11:09.233299 2704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [sys.catalog]: This master's current role is: LEADER
I20260926 02:11:09.235695 2705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f42205c49a844fc838a25f56c408269. Latest consensus state: current_term: 2 leader_uuid: "2f42205c49a844fc838a25f56c408269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f42205c49a844fc838a25f56c408269" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44923 } } }
I20260926 02:11:09.236117 2705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [sys.catalog]: This master's current role is: LEADER
I20260926 02:11:09.238049 2709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:11:09.248142 2709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:11:09.249879 2709 catalog_manager.cc:1259] Loaded cluster ID: 9f770d8049af48198830e14f9ddd7c03
I20260926 02:11:09.250018 2709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:11:09.254596 2709 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:11:09.256577 2709 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269: Loaded TSK: 0
I20260926 02:11:09.295652 2709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269: Generated new TSK 1
I20260926 02:11:09.296090 2709 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 02:11:09.551105 2701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:11:09.551414 2701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:11:09.551580 2701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:11:09.559468 2701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:11:09.559603 2701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:11:09.559792 2701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:11:09.570926 2701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:43893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=34625
--tserver_master_addrs=127.18.159.190:44923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:11:09.572389 2701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:11:09.573731 2701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:11:09.589697 2727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:09.589622 2726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:09.589656 2729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:11:09.591387 2701 server_base.cc:1034] running on GCE node
I20260926 02:11:09.592280 2701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:11:09.593652 2701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:11:09.594938 2701 hybrid_clock.cc:648] HybridClock initialized: now 1790388669594851 us; error 79 us; skew 500 ppm
I20260926 02:11:09.595335 2701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:11:09 dist-test-slave-03rn krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388669, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:11:09.599618 2701 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:11:09.601480 2701 webserver.cc:460] Webserver started at http://127.18.159.129:34625/ using document root <none> and password file <none>
I20260926 02:11:09.602241 2701 fs_manager.cc:359] Metadata directory not provided
I20260926 02:11:09.602401 2701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:11:09.608438 2701 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:11:09.611541 2736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:09.613309 2701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:11:09.613528 2701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "79231b61c8954168950f91e8d03dd1f3"
format_stamp: "Formatted at 2026-09-26 02:10:46 on dist-test-slave-03rn"
server_key: "155e736d0429762f64ba6fe220c129f4"
server_key_iv: "3faeabb531c35e2ba0c4db64188b873f"
server_key_version: "encryptionkey@0"
I20260926 02:11:09.614744 2701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:11:09.632408 2701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:11:09.637048 2701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:11:09.637363 2701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:11:09.638577 2701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:11:09.640281 2701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:11:09.640403 2701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:09.640537 2701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:11:09.640626 2701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:09.678159 2701 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:43893
I20260926 02:11:09.678221 2848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:43893 every 8 connection(s)
I20260926 02:11:09.679741 2701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:11:09.683486 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 2701
I20260926 02:11:09.684271 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:34015
--local_ip_for_outbound_sockets=127.18.159.130
--tserver_master_addrs=127.18.159.190:44923
--webserver_port=42565
--webserver_interface=127.18.159.130
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:11:09 dist-test-slave-03rn krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388669, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:11:09.712702 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:09.683200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:34387)
0926 02:11:09.684185 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:09.684241 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:09.686199 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:09.688915 (+ 2716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:09.688930 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:09.689870 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:11:09.690813 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:09.690846 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:09.693629 (+ 2783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:09.693634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:09.695037 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:09.695054 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:09.696903 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:09.696908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:09.697308 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:09.698099 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:11:09.698124 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:11:09.700954 (+ 2830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:11:09.706649 (+ 5695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:11:09.706658 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:11:09.706673 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:11:09.707240 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:11:09.708005 (+ 765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:11:09.708010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:11:09.708012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:11:09.708362 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:11:09.709663 (+ 1301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:11:09.709670 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:11:09.710023 (+ 353us) client_negotiation.cc:769] Sending connection context
0926 02:11:09.710478 (+ 455us) client_negotiation.cc:240] Negotiation successful
0926 02:11:09.711100 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":412,"threads_started":1}
I20260926 02:11:09.712754 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:09.682854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:34387 (local address 127.18.159.190:44923)
0926 02:11:09.683643 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:09.683665 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:09.684632 (+ 967us) server_negotiation.cc:407] Connection header received
0926 02:11:09.686928 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:09.686958 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:09.687510 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:09.687994 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:09.691132 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:09.693382 (+ 2250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:09.695250 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:09.696589 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:09.696832 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:09.701373 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:09.701389 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:09.701416 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:09.701477 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:09.706318 (+ 4841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:09.707472 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:09.707487 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:09.707500 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:09.707800 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:09.708561 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:09.708570 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:09.708577 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:09.709434 (+ 857us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:09.709680 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:09.710646 (+ 966us) server_negotiation.cc:299] Negotiation successful
0926 02:11:09.711221 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":329,"threads_started":1}
I20260926 02:11:09.716614 2849 heartbeater.cc:347] Connected to a master server at 127.18.159.190:44923
I20260926 02:11:09.716980 2849 heartbeater.cc:464] Registering TS with master...
I20260926 02:11:09.717756 2849 heartbeater.cc:511] Master 127.18.159.190:44923 requested a full tablet report, sending...
I20260926 02:11:09.721639 2664 ts_manager.cc:194] Registered new tserver with Master: 79231b61c8954168950f91e8d03dd1f3 (127.18.159.129:43893)
I20260926 02:11:09.723220 2664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:34387
W20260926 02:11:10.046507 2853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:11:10.046886 2853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:11:10.047238 2853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:11:10.055167 2853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:11:10.055352 2853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:11:10.055543 2853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:11:10.066032 2853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:34015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=42565
--tserver_master_addrs=127.18.159.190:44923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:11:10.067514 2853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:11:10.068979 2853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:11:10.083595 2858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:10.083509 2859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:10.083475 2861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:11:10.084443 2853 server_base.cc:1034] running on GCE node
I20260926 02:11:10.085167 2853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:11:10.086328 2853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:11:10.087529 2853 hybrid_clock.cc:648] HybridClock initialized: now 1790388670087456 us; error 74 us; skew 500 ppm
I20260926 02:11:10.087889 2853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:11:10 dist-test-slave-03rn krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388670, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:11:10.091457 2853 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:11:10.093256 2853 webserver.cc:460] Webserver started at http://127.18.159.130:42565/ using document root <none> and password file <none>
I20260926 02:11:10.094033 2853 fs_manager.cc:359] Metadata directory not provided
I20260926 02:11:10.094203 2853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:11:10.100669 2853 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 02:11:10.104228 2868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:10.105861 2853 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:11:10.106114 2853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "4544ec3626644f498d99b4f9408d4eae"
format_stamp: "Formatted at 2026-09-26 02:10:47 on dist-test-slave-03rn"
server_key: "cf7875657dcbead532a0ee6fd496df76"
server_key_iv: "5e27c7ea240aee946dff8a6b1114ab60"
server_key_version: "encryptionkey@0"
I20260926 02:11:10.107235 2853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:11:10.131583 2853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:11:10.136551 2853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:11:10.136863 2853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:11:10.137956 2853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:11:10.139675 2853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:11:10.139797 2853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:10.139952 2853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:11:10.140044 2853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:10.178544 2853 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:34015
I20260926 02:11:10.178610 2980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:34015 every 8 connection(s)
I20260926 02:11:10.180124 2853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:11:10.185971 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 2853
I20260926 02:11:10.186795 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:38029
--local_ip_for_outbound_sockets=127.18.159.131
--tserver_master_addrs=127.18.159.190:44923
--webserver_port=39463
--webserver_interface=127.18.159.131
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:11:10 dist-test-slave-03rn krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388670, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:11:10.206476 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:10.183019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:44569 (local address 127.18.159.190:44923)
0926 02:11:10.183219 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:10.183223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:10.184596 (+ 1373us) server_negotiation.cc:407] Connection header received
0926 02:11:10.187275 (+ 2679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:10.187280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:10.187372 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:10.187460 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:10.189807 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.190494 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.193428 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.193898 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.194007 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:10.198390 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:10.198401 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:10.198408 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:10.198459 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:10.202432 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:10.203583 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:10.203593 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:10.203599 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:10.203779 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:10.204347 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:10.204356 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:10.204362 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:10.204806 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:10.204968 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:10.205734 (+ 766us) server_negotiation.cc:299] Negotiation successful
0926 02:11:10.206017 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:11:10.207216 2983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:10.183537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.130:44569)
0926 02:11:10.184375 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:10.184427 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:10.186895 (+ 2468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:10.187789 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:10.187806 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:10.188693 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:11:10.189593 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:10.189622 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.190675 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.190683 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:10.193217 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:10.193240 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.194187 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.194197 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:10.194562 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:10.195422 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:11:10.195452 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:11:10.198063 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:11:10.202673 (+ 4610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:11:10.202686 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:11:10.202709 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:11:10.203370 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:11:10.203917 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:11:10.203921 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:11:10.203923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:11:10.204202 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:11:10.204934 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:11:10.204940 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:11:10.205321 (+ 381us) client_negotiation.cc:769] Sending connection context
0926 02:11:10.205568 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 02:11:10.206007 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":293,"threads_started":1}
I20260926 02:11:10.208966 2981 heartbeater.cc:347] Connected to a master server at 127.18.159.190:44923
I20260926 02:11:10.209350 2981 heartbeater.cc:464] Registering TS with master...
I20260926 02:11:10.210399 2981 heartbeater.cc:511] Master 127.18.159.190:44923 requested a full tablet report, sending...
I20260926 02:11:10.212392 2664 ts_manager.cc:194] Registered new tserver with Master: 4544ec3626644f498d99b4f9408d4eae (127.18.159.130:34015)
I20260926 02:11:10.213447 2664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:44569
W20260926 02:11:10.550751 2984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:11:10.551131 2984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:11:10.551303 2984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:11:10.558988 2984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:11:10.559175 2984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:11:10.559398 2984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:11:10.569924 2984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:38029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=39463
--tserver_master_addrs=127.18.159.190:44923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:11:10.571310 2984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:11:10.572713 2984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:11:10.586067 2989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:10.586084 2990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:10.586668 2992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:11:10.587134 2984 server_base.cc:1034] running on GCE node
I20260926 02:11:10.587764 2984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:11:10.588848 2984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:11:10.590088 2984 hybrid_clock.cc:648] HybridClock initialized: now 1790388670590046 us; error 37 us; skew 500 ppm
I20260926 02:11:10.590402 2984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:11:10 dist-test-slave-03rn krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388670, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:11:10.593827 2984 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:11:10.595506 2984 webserver.cc:460] Webserver started at http://127.18.159.131:39463/ using document root <none> and password file <none>
I20260926 02:11:10.596340 2984 fs_manager.cc:359] Metadata directory not provided
I20260926 02:11:10.596508 2984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:11:10.602540 2984 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 02:11:10.605783 2999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:10.607246 2984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 02:11:10.607457 2984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "8ebf3a462272420e821e8b9e2f8798ae"
format_stamp: "Formatted at 2026-09-26 02:10:47 on dist-test-slave-03rn"
server_key: "978cacf6acdcc7c73fae1fcb68955816"
server_key_iv: "8ed615e48ee58f20ac8b64f712bee89e"
server_key_version: "encryptionkey@0"
I20260926 02:11:10.608605 2984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:11:10.634713 2984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:11:10.639407 2984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:11:10.639711 2984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:11:10.640823 2984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:11:10.642283 2984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:11:10.642424 2984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:10.642566 2984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:11:10.642655 2984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:10.678757 2984 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:38029
I20260926 02:11:10.678823 3111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:38029 every 8 connection(s)
I20260926 02:11:10.680485 2984 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:11:10.688621 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 2984
Sep 26 02:11:10 dist-test-slave-03rn krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388670, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:11:10.705559 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:10.683610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:45615 (local address 127.18.159.190:44923)
0926 02:11:10.683816 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:10.683821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:10.685171 (+ 1350us) server_negotiation.cc:407] Connection header received
0926 02:11:10.687273 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:10.687280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:10.687379 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:10.687531 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:10.689593 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.690683 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.692458 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.692993 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.693156 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:10.697194 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:10.697202 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:10.697208 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:10.697256 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:10.701062 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:10.702307 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:10.702313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:10.702315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:10.702511 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:10.703070 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:10.703078 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:10.703081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:10.703528 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:10.703755 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:10.704850 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0926 02:11:10.705132 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":76}
I20260926 02:11:10.706275 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:10.689656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58376 (local address 127.18.159.190:44923)
0926 02:11:10.690480 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:10.690488 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:10.690508 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:11:10.690565 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:10.690570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:10.690645 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:10.690769 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:10.692352 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.693236 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.694843 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.695284 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.695434 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:10.699108 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:10.699115 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:10.699118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:10.699164 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:10.702537 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:10.703657 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:10.703663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:10.703667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:10.703811 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:10.704315 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:10.704320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:10.704323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:10.704674 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:10.704954 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:10.705390 (+ 436us) server_negotiation.cc:299] Negotiation successful
0926 02:11:10.705631 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":704,"thread_start_us":539,"threads_started":1}
I20260926 02:11:10.706812 3114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:10.684114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.131:45615)
0926 02:11:10.684982 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:10.685020 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:10.686967 (+ 1947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:10.687869 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:10.687879 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:10.688561 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:11:10.689315 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:10.689346 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.690903 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.690908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:10.692286 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:10.692299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.693249 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.693257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:10.693558 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:10.694335 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:11:10.694359 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:11:10.696868 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:11:10.701282 (+ 4414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:11:10.701296 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:11:10.701320 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:11:10.702092 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:11:10.702660 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:11:10.702666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:11:10.702669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:11:10.702919 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:11:10.703685 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:11:10.703695 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:11:10.704268 (+ 573us) client_negotiation.cc:769] Sending connection context
0926 02:11:10.704686 (+ 418us) client_negotiation.cc:240] Negotiation successful
0926 02:11:10.705276 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":340,"threads_started":1}
I20260926 02:11:10.708714 3112 heartbeater.cc:347] Connected to a master server at 127.18.159.190:44923
I20260926 02:11:10.709626 3112 heartbeater.cc:464] Registering TS with master...
I20260926 02:11:10.710660 3112 heartbeater.cc:511] Master 127.18.159.190:44923 requested a full tablet report, sending...
I20260926 02:11:10.712563 2664 ts_manager.cc:194] Registered new tserver with Master: 8ebf3a462272420e821e8b9e2f8798ae (127.18.159.131:38029)
I20260926 02:11:10.713460 2664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:45615
I20260926 02:11:10.724015 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:11:10.727039 2849 heartbeater.cc:503] Master 127.18.159.190:44923 was elected leader, sending a full tablet report...
I20260926 02:11:10.731849 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:10.725234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58380 (local address 127.18.159.190:44923)
0926 02:11:10.725574 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:10.725579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:10.725958 (+ 379us) server_negotiation.cc:407] Connection header received
0926 02:11:10.726128 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:10.726133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:10.726285 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:10.726473 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:11:10.727058 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.728018 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.729598 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.730076 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.730254 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:10.730892 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:11:10.731120 (+ 228us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 02:11:10.731532 (+ 412us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58380: token expired
Metrics: {"server-negotiator.queue_time_us":196}
W20260926 02:11:10.732015 3116 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58380: token expired
W20260926 02:11:10.732568 2585 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.159.190:44923)
I20260926 02:11:10.747337 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:10.733145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58392 (local address 127.18.159.190:44923)
0926 02:11:10.733338 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:10.733344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:10.733365 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:11:10.733576 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:10.733583 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:10.733665 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:10.733772 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:10.735259 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.736021 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.737613 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.738047 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.738206 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:10.741407 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:10.741419 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:10.741423 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:10.741464 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:10.744151 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:10.745038 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:10.745045 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:10.745048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:10.745216 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:10.745657 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:10.745663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:10.745666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:10.746028 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:10.746370 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:10.746699 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 02:11:10.746965 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:11:10.751750 2664 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:11:10.759687 2664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:11:10.788108 3122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:10.773472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:38029 (local address 127.0.0.1:37848)
0926 02:11:10.774431 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:10.774486 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:10.774741 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:10.777324 (+ 2583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:10.777336 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:10.777385 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:11:10.777887 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:10.777907 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.780355 (+ 2448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.780361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:10.783584 (+ 3223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:10.783610 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.786934 (+ 3324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.786944 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:10.787247 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:10.787294 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 02:11:10.787478 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 02:11:10.787716 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":11739,"threads_started":1}
I20260926 02:11:10.789144 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:10.773525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37848 (local address 127.18.159.131:38029)
0926 02:11:10.776145 (+ 2620us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:10.776179 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:10.776216 (+ 37us) server_negotiation.cc:407] Connection header received
0926 02:11:10.776335 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:10.776344 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:10.776663 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:10.776905 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:11:10.778185 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.780150 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.783839 (+ 3689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.785764 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.786300 (+ 536us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:10.786574 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:10.788681 (+ 2107us) server_negotiation.cc:299] Negotiation successful
0926 02:11:10.788894 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2403,"thread_start_us":3618,"threads_started":1}
I20260926 02:11:10.793386 3047 tablet_service.cc:1467] Processing CreateTablet for tablet 7857dc6a10964493ab3c9f34ba851936 (DEFAULT_TABLE table=test-table [id=79c24aefc2d94ec882ee40ae70f97503]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:11:10.794971 3123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:10.773494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:34015 (local address 127.0.0.1:46892)
0926 02:11:10.785525 (+ 12031us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:10.785551 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:10.785815 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:10.786817 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:10.786825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:10.786867 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:11:10.787512 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:10.787528 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.789658 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.789665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:10.791740 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:10.791756 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.794011 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.794018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:10.794202 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:10.794238 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:11:10.794339 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 02:11:10.794493 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11880,"mutex_wait_us":1,"thread_start_us":1394,"threads_started":1}
I20260926 02:11:10.795215 3047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7857dc6a10964493ab3c9f34ba851936. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:11:10.795397 3125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:10.774146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46892 (local address 127.18.159.130:34015)
0926 02:11:10.784740 (+ 10594us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:10.784753 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:10.785681 (+ 928us) server_negotiation.cc:407] Connection header received
0926 02:11:10.786091 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:10.786103 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:10.786398 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:10.786546 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:11:10.787730 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.789486 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.791943 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.793448 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.793661 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:10.793850 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:10.794882 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0926 02:11:10.795069 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10313,"thread_start_us":10596,"threads_started":1}
I20260926 02:11:10.795904 3127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:10.776527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54468 (local address 127.18.159.129:43893)
0926 02:11:10.786022 (+ 9495us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:10.786031 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:10.787921 (+ 1890us) server_negotiation.cc:407] Connection header received
0926 02:11:10.788041 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:10.788050 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:10.788358 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:10.788570 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:11:10.789346 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.790588 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.792861 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.794315 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.794562 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:10.794754 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:10.795380 (+ 626us) server_negotiation.cc:299] Negotiation successful
0926 02:11:10.795576 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9257,"thread_start_us":9123,"threads_started":1}
I20260926 02:11:10.795929 3126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:10.775911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:43893 (local address 127.0.0.1:54468)
0926 02:11:10.787632 (+ 11721us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:10.787652 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:10.787770 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:10.788826 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:10.788831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:10.788866 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:11:10.789152 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:10.789165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.790742 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.790748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:10.792653 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:10.792670 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:10.794585 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:10.794592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:10.794770 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:10.794819 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 02:11:10.794911 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 02:11:10.795346 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11551,"mutex_wait_us":1,"thread_start_us":11082,"threads_started":1}
I20260926 02:11:10.799072 2916 tablet_service.cc:1467] Processing CreateTablet for tablet 7857dc6a10964493ab3c9f34ba851936 (DEFAULT_TABLE table=test-table [id=79c24aefc2d94ec882ee40ae70f97503]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:11:10.799736 2784 tablet_service.cc:1467] Processing CreateTablet for tablet 7857dc6a10964493ab3c9f34ba851936 (DEFAULT_TABLE table=test-table [id=79c24aefc2d94ec882ee40ae70f97503]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:11:10.800702 2916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7857dc6a10964493ab3c9f34ba851936. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:11:10.801278 2784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7857dc6a10964493ab3c9f34ba851936. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:11:10.813326 3128 tablet_bootstrap.cc:492] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae: Bootstrap starting.
I20260926 02:11:10.815343 3129 tablet_bootstrap.cc:492] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae: Bootstrap starting.
I20260926 02:11:10.817834 3130 tablet_bootstrap.cc:492] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3: Bootstrap starting.
I20260926 02:11:10.819268 3129 tablet_bootstrap.cc:654] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae: Neither blocks nor log segments found. Creating new log.
I20260926 02:11:10.819814 3128 tablet_bootstrap.cc:654] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae: Neither blocks nor log segments found. Creating new log.
I20260926 02:11:10.820603 3129 log.cc:824] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae: Log is configured to *not* fsync() on all Append() calls
I20260926 02:11:10.821094 3128 log.cc:824] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae: Log is configured to *not* fsync() on all Append() calls
I20260926 02:11:10.821714 3130 tablet_bootstrap.cc:654] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3: Neither blocks nor log segments found. Creating new log.
I20260926 02:11:10.823201 3130 log.cc:824] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3: Log is configured to *not* fsync() on all Append() calls
I20260926 02:11:10.824628 3129 tablet_bootstrap.cc:492] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae: No bootstrap required, opened a new log
I20260926 02:11:10.824698 3128 tablet_bootstrap.cc:492] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae: No bootstrap required, opened a new log
I20260926 02:11:10.824951 3129 ts_tablet_manager.cc:1397] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 02:11:10.824998 3128 ts_tablet_manager.cc:1397] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260926 02:11:10.826786 3130 tablet_bootstrap.cc:492] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3: No bootstrap required, opened a new log
I20260926 02:11:10.827095 3130 ts_tablet_manager.cc:1397] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260926 02:11:10.833364 3129 raft_consensus.cc:348] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } }
I20260926 02:11:10.833841 3129 raft_consensus.cc:374] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:11:10.833947 3129 raft_consensus.cc:729] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4544ec3626644f498d99b4f9408d4eae, State: Initialized, Role: FOLLOWER
I20260926 02:11:10.834661 3129 consensus_queue.cc:260] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } }
I20260926 02:11:10.835882 2981 heartbeater.cc:503] Master 127.18.159.190:44923 was elected leader, sending a full tablet report...
I20260926 02:11:10.836118 3128 raft_consensus.cc:348] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } }
I20260926 02:11:10.836637 3128 raft_consensus.cc:374] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:11:10.836707 3129 ts_tablet_manager.cc:1428] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260926 02:11:10.836726 3128 raft_consensus.cc:729] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ebf3a462272420e821e8b9e2f8798ae, State: Initialized, Role: FOLLOWER
I20260926 02:11:10.837504 3128 consensus_queue.cc:260] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } }
I20260926 02:11:10.838773 3112 heartbeater.cc:503] Master 127.18.159.190:44923 was elected leader, sending a full tablet report...
I20260926 02:11:10.838630 3130 raft_consensus.cc:348] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } }
I20260926 02:11:10.839099 3130 raft_consensus.cc:374] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:11:10.839196 3130 raft_consensus.cc:729] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79231b61c8954168950f91e8d03dd1f3, State: Initialized, Role: FOLLOWER
I20260926 02:11:10.839592 3128 ts_tablet_manager.cc:1428] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260926 02:11:10.839998 3130 consensus_queue.cc:260] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } }
I20260926 02:11:10.843528 3130 ts_tablet_manager.cc:1428] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
W20260926 02:11:10.932430 2982 tablet.cc:2369] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:11:10.932389 3113 tablet.cc:2369] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:11:10.932387 2850 tablet.cc:2369] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:11:11.030479 3135 raft_consensus.cc:482] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:11:11.030781 3135 raft_consensus.cc:504] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } }
I20260926 02:11:11.032671 3135 leader_election.cc:288] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4544ec3626644f498d99b4f9408d4eae (127.18.159.130:34015), 79231b61c8954168950f91e8d03dd1f3 (127.18.159.129:43893)
I20260926 02:11:11.038631 3114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:11.033010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:34015 (local address 127.18.159.131:54497)
0926 02:11:11.033238 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:11.033259 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:11.033403 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:11.033901 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:11.033905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:11.033948 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:11:11.034135 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:11.034143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:11.035178 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:11.035187 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:11.036588 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:11.036609 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:11.037791 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:11.037797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:11.037995 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:11.038032 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 02:11:11.038146 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 02:11:11.038296 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 02:11:11.038622 3125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:11.033195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:54497 (local address 127.18.159.130:34015)
0926 02:11:11.033439 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:11.033448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:11.033501 (+ 53us) server_negotiation.cc:407] Connection header received
0926 02:11:11.033614 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:11.033623 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:11.033719 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:11.033819 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:11:11.034290 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:11.035025 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:11.036781 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:11.037615 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:11.037833 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:11.037958 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:11.038200 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 02:11:11.038342 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 02:11:11.039659 3127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:11.033632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:56281 (local address 127.18.159.129:43893)
0926 02:11:11.033850 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:11.033855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:11.034081 (+ 226us) server_negotiation.cc:407] Connection header received
0926 02:11:11.034271 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:11.034276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:11.034353 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:11.034499 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:11:11.035046 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:11.035737 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:11.037537 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:11.038472 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:11.038665 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:11.038752 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:11.039249 (+ 497us) server_negotiation.cc:299] Negotiation successful
0926 02:11:11.039415 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 02:11:11.039718 3137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:11.033360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:43893 (local address 127.18.159.131:56281)
0926 02:11:11.033969 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:11.033982 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:11.034130 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:11.034512 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:11.034516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:11.034536 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:11:11.034871 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:11.034879 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:11.035880 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:11.035886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:11.037352 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:11.037364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:11.038756 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:11.038763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:11.039076 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:11.039114 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 02:11:11.039212 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 02:11:11.039358 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":305,"threads_started":1}
I20260926 02:11:11.039901 2936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7857dc6a10964493ab3c9f34ba851936" candidate_uuid: "8ebf3a462272420e821e8b9e2f8798ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4544ec3626644f498d99b4f9408d4eae" is_pre_election: true
I20260926 02:11:11.040532 2936 raft_consensus.cc:2454] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ebf3a462272420e821e8b9e2f8798ae in term 0.
I20260926 02:11:11.040861 2804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7857dc6a10964493ab3c9f34ba851936" candidate_uuid: "8ebf3a462272420e821e8b9e2f8798ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79231b61c8954168950f91e8d03dd1f3" is_pre_election: true
I20260926 02:11:11.041308 3001 leader_election.cc:302] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4544ec3626644f498d99b4f9408d4eae, 8ebf3a462272420e821e8b9e2f8798ae; no voters:
I20260926 02:11:11.041481 2804 raft_consensus.cc:2454] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ebf3a462272420e821e8b9e2f8798ae in term 0.
I20260926 02:11:11.041751 3135 raft_consensus.cc:2781] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:11:11.041915 3135 raft_consensus.cc:482] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:11:11.042033 3135 raft_consensus.cc:3037] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:11:11.044447 3135 raft_consensus.cc:504] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } }
I20260926 02:11:11.045100 3135 leader_election.cc:288] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae [CANDIDATE]: Term 1 election: Requested vote from peers 4544ec3626644f498d99b4f9408d4eae (127.18.159.130:34015), 79231b61c8954168950f91e8d03dd1f3 (127.18.159.129:43893)
I20260926 02:11:11.045737 2804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7857dc6a10964493ab3c9f34ba851936" candidate_uuid: "8ebf3a462272420e821e8b9e2f8798ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79231b61c8954168950f91e8d03dd1f3"
I20260926 02:11:11.045733 2936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7857dc6a10964493ab3c9f34ba851936" candidate_uuid: "8ebf3a462272420e821e8b9e2f8798ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4544ec3626644f498d99b4f9408d4eae"
I20260926 02:11:11.045979 2936 raft_consensus.cc:3037] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:11:11.045997 2804 raft_consensus.cc:3037] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:11:11.048346 2804 raft_consensus.cc:2454] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ebf3a462272420e821e8b9e2f8798ae in term 1.
I20260926 02:11:11.048344 2936 raft_consensus.cc:2454] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ebf3a462272420e821e8b9e2f8798ae in term 1.
I20260926 02:11:11.048838 3001 leader_election.cc:302] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4544ec3626644f498d99b4f9408d4eae, 8ebf3a462272420e821e8b9e2f8798ae; no voters:
I20260926 02:11:11.049235 3135 raft_consensus.cc:2781] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:11:11.049742 3135 raft_consensus.cc:686] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae [term 1 LEADER]: Becoming Leader. State: Replica: 8ebf3a462272420e821e8b9e2f8798ae, State: Running, Role: LEADER
I20260926 02:11:11.050261 3135 consensus_queue.cc:237] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } }
I20260926 02:11:11.055187 2664 catalog_manager.cc:5495] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ebf3a462272420e821e8b9e2f8798ae (127.18.159.131). New cstate: current_term: 1 leader_uuid: "8ebf3a462272420e821e8b9e2f8798ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } health_report { overall_health: UNKNOWN } } }
I20260926 02:11:11.087934 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:11.081823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37860 (local address 127.18.159.131:38029)
0926 02:11:11.082083 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:11.082091 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:11.082195 (+ 104us) server_negotiation.cc:407] Connection header received
0926 02:11:11.082481 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:11.082486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:11.082588 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:11.082761 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:11:11.083160 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:11.083915 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:11.085456 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:11.085939 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:11.086085 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:11.086695 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:11:11.087294 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:11:11.087470 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:11.087560 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 02:11:11.087653 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 02:11:11.097146 2936 raft_consensus.cc:1261] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae [term 1 FOLLOWER]: Refusing update from remote peer 8ebf3a462272420e821e8b9e2f8798ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:11:11.097193 2804 raft_consensus.cc:1261] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Refusing update from remote peer 8ebf3a462272420e821e8b9e2f8798ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:11:11.098385 3138 consensus_queue.cc:1035] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:11:11.098814 3135 consensus_queue.cc:1035] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:11:11.111387 3141 mvcc.cc:205] Tried to move back new op lower bound from 7333431996801974272 to 7333431996625924096. Current Snapshot: MvccSnapshot[applied={T|T < 7333431996801974272}]
I20260926 02:11:11.114630 3142 mvcc.cc:205] Tried to move back new op lower bound from 7333431996801974272 to 7333431996625924096. Current Snapshot: MvccSnapshot[applied={T|T < 7333431996801974272}]
I20260926 02:11:11.117286 3143 mvcc.cc:205] Tried to move back new op lower bound from 7333431996801974272 to 7333431996625924096. Current Snapshot: MvccSnapshot[applied={T|T < 7333431996801974272}]
I20260926 02:11:11.128546 2663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:11:11.128906 2663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:11:11.133351 2663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269: Sending DeleteTablet for 3 replicas of tablet 7857dc6a10964493ab3c9f34ba851936
I20260926 02:11:11.134884 2916 tablet_service.cc:1514] Processing DeleteTablet for tablet 7857dc6a10964493ab3c9f34ba851936 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:11:11 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:46892
I20260926 02:11:11.134886 2784 tablet_service.cc:1514] Processing DeleteTablet for tablet 7857dc6a10964493ab3c9f34ba851936 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:11:11 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:54468
I20260926 02:11:11.134896 3047 tablet_service.cc:1514] Processing DeleteTablet for tablet 7857dc6a10964493ab3c9f34ba851936 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:11:11 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:37848
I20260926 02:11:11.136351 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 2701
I20260926 02:11:11.147563 3154 tablet_replica.cc:330] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae: stopping tablet replica
I20260926 02:11:11.148322 3154 raft_consensus.cc:2229] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:11:11.149210 3154 raft_consensus.cc:2258] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:11:11.151800 3154 ts_tablet_manager.cc:1905] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:11:11.173977 3154 ts_tablet_manager.cc:1918] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:11:11.175474 3127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:11.138438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54484 (local address 127.18.159.129:43893)
0926 02:11:11.138786 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:11.138795 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:11.138821 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:11:11.138905 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:11.138913 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:11.139023 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:11.139154 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:11.142036 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:11.143709 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:11.149105 (+ 5396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:11.150160 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:11.150314 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:11.160782 (+ 10468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:11.160814 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:11.160843 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:11.160926 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:11.167039 (+ 6113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:11.169553 (+ 2514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:11.169570 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:11.169578 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:11.169886 (+ 308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:11.170923 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:11.170932 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:11.170938 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:11.171663 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:11.172099 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:11.174439 (+ 2340us) server_negotiation.cc:299] Negotiation successful
0926 02:11:11.174839 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260926 02:11:11.176524 2844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:11:11.176903 3154 log.cc:1196] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/7857dc6a10964493ab3c9f34ba851936
I20260926 02:11:11.196381 3154 ts_tablet_manager.cc:1939] T 7857dc6a10964493ab3c9f34ba851936 P 8ebf3a462272420e821e8b9e2f8798ae: Deleting consensus metadata
I20260926 02:11:11.198863 2648 catalog_manager.cc:4841] TS 8ebf3a462272420e821e8b9e2f8798ae (127.18.159.131:38029): tablet 7857dc6a10964493ab3c9f34ba851936 (table test-table [id=79c24aefc2d94ec882ee40ae70f97503]) successfully deleted
I20260926 02:11:11.202507 3153 tablet_replica.cc:330] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae: stopping tablet replica
I20260926 02:11:11.203177 3153 raft_consensus.cc:2229] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:11:11.204852 3153 raft_consensus.cc:2258] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:11:11.207772 3153 ts_tablet_manager.cc:1905] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:11:11.216493 3153 ts_tablet_manager.cc:1918] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:11:11.216713 3153 log.cc:1196] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/7857dc6a10964493ab3c9f34ba851936
I20260926 02:11:11.217284 3153 ts_tablet_manager.cc:1939] T 7857dc6a10964493ab3c9f34ba851936 P 4544ec3626644f498d99b4f9408d4eae: Deleting consensus metadata
I20260926 02:11:11.219344 2649 catalog_manager.cc:4841] TS 4544ec3626644f498d99b4f9408d4eae (127.18.159.130:34015): tablet 7857dc6a10964493ab3c9f34ba851936 (table test-table [id=79c24aefc2d94ec882ee40ae70f97503]) successfully deleted
I20260926 02:11:11.317656 3152 tablet_replica.cc:330] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3: stopping tablet replica
I20260926 02:11:11.318212 3152 raft_consensus.cc:2229] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:11:11.318353 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 2701
I20260926 02:11:11.335904 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 2853
W20260926 02:11:11.335918 2649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.159.129:43893 (error 108)
I20260926 02:11:11.338533 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:11.337585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:43893 (local address 127.0.0.1:54500)
0926 02:11:11.338167 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:11.338304 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":361,"threads_started":1}
W20260926 02:11:11.338881 2649 catalog_manager.cc:4584] TS 79231b61c8954168950f91e8d03dd1f3 (127.18.159.129:43893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7857dc6a10964493ab3c9f34ba851936: Network error: Client connection negotiation failed: client connection to 127.18.159.129:43893: connect: Connection refused (error 111)
I20260926 02:11:11.354337 3125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:11.337147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46908 (local address 127.18.159.130:34015)
0926 02:11:11.337390 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:11.337400 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:11.337427 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:11:11.337496 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:11.337501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:11.337588 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:11.337711 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:11.339559 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:11.340776 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:11.342330 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:11.343004 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:11.343163 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:11.346439 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:11.346462 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:11.346482 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:11.346540 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:11.350891 (+ 4351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:11.351826 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:11.351837 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:11.351841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:11.352037 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:11.352531 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:11.352536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:11.352540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:11.353062 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:11.353378 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:11.353691 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 02:11:11.353948 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 02:11:11.355101 2976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:11:11.392531 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:11.392015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:43893 (local address 127.0.0.1:54516)
0926 02:11:11.392245 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:11.392347 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 02:11:11.393908 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:11.393517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:43893 (local address 127.0.0.1:54526)
0926 02:11:11.393679 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:11.393758 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 02:11:11.454574 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:11.454092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:43893 (local address 127.0.0.1:54528)
0926 02:11:11.454324 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:11.454411 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 02:11:11.455727 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:11.455342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:43893 (local address 127.0.0.1:54534)
0926 02:11:11.455510 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:11.455582 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 02:11:11.504071 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 2853
I20260926 02:11:11.522737 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 2984
I20260926 02:11:11.536613 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:11.536068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:43893 (local address 127.0.0.1:54540)
0926 02:11:11.536309 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:11.536458 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260926 02:11:11.538138 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:11.537373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:43893 (local address 127.0.0.1:54542)
0926 02:11:11.537698 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:11.537851 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260926 02:11:11.548576 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:11.530528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37874 (local address 127.18.159.131:38029)
0926 02:11:11.530783 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:11.530790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:11.530811 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:11:11.530877 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:11.530889 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:11.530970 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:11.531076 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:11.533304 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:11.534241 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:11.536344 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:11.537090 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:11.537255 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:11.541656 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:11.541671 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:11.541686 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:11.541742 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:11.545017 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:11.546135 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:11.546144 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:11.546146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:11.546296 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:11.546820 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:11.546825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:11.546827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:11.547215 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:11.547505 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:11.547907 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 02:11:11.548118 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 02:11:11.549497 3107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:11:11.680354 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 2984
I20260926 02:11:11.700850 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 2631
I20260926 02:11:11.702092 2694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:11:11.819834 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:11.819214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:43893 (local address 127.0.0.1:54558)
0926 02:11:11.819465 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:11.819641 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 02:11:11.819891 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 2631
I20260926 02:11:31.842598 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:44923
--webserver_interface=127.18.159.190
--webserver_port=34987
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:44923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:11:32.187448 3161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:11:32.187793 3161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:11:32.187947 3161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:11:32.195534 3161 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 02:11:32.195647 3161 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 02:11:32.195685 3161 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 02:11:32.195781 3161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:11:32.195881 3161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:11:32.196000 3161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:11:32.196092 3161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:11:32.196193 3161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:11:32.206571 3161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:44923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:44923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=34987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:11:32.207970 3161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:11:32.209426 3161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:11:32.222306 3167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:32.222369 3166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:32.222369 3169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:11:32.223053 3161 server_base.cc:1034] running on GCE node
I20260926 02:11:32.224018 3161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:11:32.225602 3161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:11:32.226840 3161 hybrid_clock.cc:648] HybridClock initialized: now 1790388692226752 us; error 83 us; skew 500 ppm
I20260926 02:11:32.227211 3161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:11:32 dist-test-slave-03rn krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388692, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:11:32.235248 3161 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:11:32.237051 3161 webserver.cc:460] Webserver started at http://127.18.159.190:34987/ using document root <none> and password file <none>
I20260926 02:11:32.237855 3161 fs_manager.cc:359] Metadata directory not provided
I20260926 02:11:32.237952 3161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:11:32.243968 3161 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 02:11:32.247231 3176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:32.248919 3161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:11:32.249132 3161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "2f42205c49a844fc838a25f56c408269"
format_stamp: "Formatted at 2026-09-26 02:10:45 on dist-test-slave-03rn"
server_key: "3066a25d6c037c7866a55bdb4fd9c090"
server_key_iv: "1135d650f88efde7a9dd5526ef29b1b6"
server_key_version: "encryptionkey@0"
I20260926 02:11:32.250170 3161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:11:32.270129 3161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:11:32.275000 3161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:11:32.275332 3161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:11:32.295588 3161 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:44923
I20260926 02:11:32.295619 3227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:44923 every 8 connection(s)
I20260926 02:11:32.297298 3161 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:11:32.298023 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 3161
I20260926 02:11:32.299048 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:43893
--local_ip_for_outbound_sockets=127.18.159.129
--tserver_master_addrs=127.18.159.190:44923
--webserver_port=34625
--webserver_interface=127.18.159.129
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:11:32.304800 3228 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:11:32.308204 3228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269: Bootstrap starting.
I20260926 02:11:32.329631 3228 log.cc:824] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269: Log is configured to *not* fsync() on all Append() calls
I20260926 02:11:32.345651 3228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:11:32.346251 3228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269: Bootstrap complete.
I20260926 02:11:32.355767 3228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f42205c49a844fc838a25f56c408269" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44923 } }
I20260926 02:11:32.356808 3228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f42205c49a844fc838a25f56c408269, State: Initialized, Role: FOLLOWER
I20260926 02:11:32.357486 3228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f42205c49a844fc838a25f56c408269" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44923 } }
I20260926 02:11:32.357741 3228 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:11:32.357874 3228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:11:32.358058 3228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 2 FOLLOWER]: Advancing to term 3
I20260926 02:11:32.362089 3228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f42205c49a844fc838a25f56c408269" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44923 } }
I20260926 02:11:32.362639 3228 leader_election.cc:302] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f42205c49a844fc838a25f56c408269; no voters:
I20260926 02:11:32.363340 3228 leader_election.cc:288] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 02:11:32.363482 3232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 02:11:32.364727 3232 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [term 3 LEADER]: Becoming Leader. State: Replica: 2f42205c49a844fc838a25f56c408269, State: Running, Role: LEADER
I20260926 02:11:32.365391 3232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f42205c49a844fc838a25f56c408269" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44923 } }
I20260926 02:11:32.365650 3228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:11:32.372255 3233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2f42205c49a844fc838a25f56c408269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f42205c49a844fc838a25f56c408269" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44923 } } }
I20260926 02:11:32.372766 3233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [sys.catalog]: This master's current role is: LEADER
I20260926 02:11:32.373198 3234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f42205c49a844fc838a25f56c408269. Latest consensus state: current_term: 3 leader_uuid: "2f42205c49a844fc838a25f56c408269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f42205c49a844fc838a25f56c408269" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44923 } } }
I20260926 02:11:32.373566 3234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269 [sys.catalog]: This master's current role is: LEADER
I20260926 02:11:32.377454 3239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:11:32.388531 3239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:11:32.390759 3239 catalog_manager.cc:1259] Loaded cluster ID: 9f770d8049af48198830e14f9ddd7c03
I20260926 02:11:32.390995 3239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:11:32.396773 3239 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:11:32.398177 3239 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269: Loaded TSK: 0
I20260926 02:11:32.398355 3239 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2f42205c49a844fc838a25f56c408269: Loaded TSK: 1
F20260926 02:11:32.399951 3239 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f73ea9359ad google::LogMessage::Fail() at ??:0
@ 0x7f73ea93888a google::LogMessage::SendToLog() at ??:0
@ 0x7f73ea9354de google::LogMessage::Flush() at ??:0
@ 0x7f73ea936a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f73ee6997d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f73ee68873e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f74003641c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f7400363783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f74003ef2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f74003ef0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f74004395e9 std::function<>::operator()() at ??:0
@ 0x7f7400367cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7400365c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f74003ed610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f74003ed439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f73ed581345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f73ed58491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f73ed584799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f73ed55feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f73f08396db start_thread at ??:0
@ 0x7f73e97cf61f clone at ??:0
W20260926 02:11:32.708685 3230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:11:32.709082 3230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:11:32.709275 3230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:11:32.717880 3230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:11:32.718065 3230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:11:32.718250 3230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:11:32.728709 3230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:43893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=34625
--tserver_master_addrs=127.18.159.190:44923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:11:32.730090 3230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:11:32.731472 3230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:11:32.746110 3276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:32.746150 3277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:32.746188 3279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:11:32.746956 3230 server_base.cc:1034] running on GCE node
I20260926 02:11:32.747717 3230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:11:32.748868 3230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:11:32.750142 3230 hybrid_clock.cc:648] HybridClock initialized: now 1790388692750097 us; error 64 us; skew 500 ppm
I20260926 02:11:32.750520 3230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:11:32 dist-test-slave-03rn krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388692, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:11:32.753837 3230 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:11:32.755496 3230 webserver.cc:460] Webserver started at http://127.18.159.129:34625/ using document root <none> and password file <none>
I20260926 02:11:32.756337 3230 fs_manager.cc:359] Metadata directory not provided
I20260926 02:11:32.756497 3230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:11:32.762303 3230 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:11:32.765726 3286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:32.767249 3230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:11:32.767467 3230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "79231b61c8954168950f91e8d03dd1f3"
format_stamp: "Formatted at 2026-09-26 02:10:46 on dist-test-slave-03rn"
server_key: "155e736d0429762f64ba6fe220c129f4"
server_key_iv: "3faeabb531c35e2ba0c4db64188b873f"
server_key_version: "encryptionkey@0"
I20260926 02:11:32.768934 3230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:11:32.796329 3230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:11:32.801012 3230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:11:32.801335 3230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:11:32.802414 3230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:11:32.804483 3293 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260926 02:11:32.810714 3230 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 02:11:32.810887 3230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260926 02:11:32.811059 3230 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 02:11:32.814741 3230 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 02:11:32.814862 3230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260926 02:11:32.815240 3293 tablet_bootstrap.cc:492] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3: Bootstrap starting.
I20260926 02:11:32.843667 3293 log.cc:824] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3: Log is configured to *not* fsync() on all Append() calls
I20260926 02:11:32.857626 3293 tablet_bootstrap.cc:492] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:11:32.858278 3293 tablet_bootstrap.cc:492] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3: Bootstrap complete.
I20260926 02:11:32.858325 3230 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:43893
I20260926 02:11:32.858361 3400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:43893 every 8 connection(s)
I20260926 02:11:32.859737 3230 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:11:32.859771 3293 ts_tablet_manager.cc:1397] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3: Time spent bootstrapping tablet: real 0.045s user 0.029s sys 0.012s
I20260926 02:11:32.865360 3403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:32.862863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:38389)
0926 02:11:32.863874 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:32.864273 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":459,"threads_started":1}
W20260926 02:11:32.866097 3401 heartbeater.cc:644] Failed to heartbeat to 127.18.159.190:44923 (0 consecutive failures): Network error: Failed to ping master at 127.18.159.190:44923: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
I20260926 02:11:32.867117 3403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:32.866738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:60785)
0926 02:11:32.866910 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:32.867003 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 02:11:32.868093 3403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:32.867744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:46061)
0926 02:11:32.867872 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:32.867947 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 02:11:32.868458 3401 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 02:11:32.868075 3293 raft_consensus.cc:348] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } }
I20260926 02:11:32.869247 3293 raft_consensus.cc:729] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79231b61c8954168950f91e8d03dd1f3, State: Initialized, Role: FOLLOWER
I20260926 02:11:32.869853 3293 consensus_queue.cc:260] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } }
I20260926 02:11:32.870044 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 3230
I20260926 02:11:32.870882 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:34015
--local_ip_for_outbound_sockets=127.18.159.130
--tserver_master_addrs=127.18.159.190:44923
--webserver_port=42565
--webserver_interface=127.18.159.130
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:11:32.871456 3293 ts_tablet_manager.cc:1428] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3: Time spent starting tablet: real 0.012s user 0.013s sys 0.001s
I20260926 02:11:32.871649 3403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:32.871161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:59519)
0926 02:11:32.871391 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:32.871490 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 02:11:32.872573 3403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:32.872219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:36191)
0926 02:11:32.872378 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:32.872456 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 02:11:33.211422 3405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:11:33.211731 3405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:11:33.211829 3405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:11:33.219581 3405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:11:33.219708 3405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:11:33.219887 3405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:11:33.230214 3405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:34015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=42565
--tserver_master_addrs=127.18.159.190:44923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:11:33.231590 3405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:11:33.233006 3405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:11:33.247640 3413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:33.247572 3411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:33.247674 3410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:11:33.254153 3405 server_base.cc:1034] running on GCE node
I20260926 02:11:33.254863 3405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:11:33.255925 3405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:11:33.257141 3405 hybrid_clock.cc:648] HybridClock initialized: now 1790388693257078 us; error 105 us; skew 500 ppm
I20260926 02:11:33.257495 3405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:11:33 dist-test-slave-03rn krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388693, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:11:33.261154 3405 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:11:33.262971 3405 webserver.cc:460] Webserver started at http://127.18.159.130:42565/ using document root <none> and password file <none>
I20260926 02:11:33.263784 3405 fs_manager.cc:359] Metadata directory not provided
I20260926 02:11:33.263952 3405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:11:33.270128 3405 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:11:33.273538 3420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:33.275300 3405 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:11:33.275522 3405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "4544ec3626644f498d99b4f9408d4eae"
format_stamp: "Formatted at 2026-09-26 02:10:47 on dist-test-slave-03rn"
server_key: "cf7875657dcbead532a0ee6fd496df76"
server_key_iv: "5e27c7ea240aee946dff8a6b1114ab60"
server_key_version: "encryptionkey@0"
I20260926 02:11:33.276723 3405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:11:33.295282 3405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:11:33.300233 3405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:11:33.300535 3405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:11:33.301640 3405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:11:33.303131 3405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:11:33.303260 3405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:33.303400 3405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:11:33.303493 3405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:33.335575 3405 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:34015
I20260926 02:11:33.335662 3534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:34015 every 8 connection(s)
I20260926 02:11:33.337083 3405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:11:33.342252 3537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:33.340253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.130:54247)
0926 02:11:33.341086 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:33.341324 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":321,"threads_started":1}
W20260926 02:11:33.343003 3535 heartbeater.cc:644] Failed to heartbeat to 127.18.159.190:44923 (0 consecutive failures): Network error: Failed to ping master at 127.18.159.190:44923: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
I20260926 02:11:33.344064 3537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:33.343639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.130:55883)
0926 02:11:33.343848 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:33.343940 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 02:11:33.345077 3537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:33.344679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.130:33733)
0926 02:11:33.344872 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:33.344952 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 02:11:33.345333 3535 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 02:11:33.346729 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 3405
I20260926 02:11:33.347409 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:38029
--local_ip_for_outbound_sockets=127.18.159.131
--tserver_master_addrs=127.18.159.190:44923
--webserver_port=39463
--webserver_interface=127.18.159.131
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:11:33.693886 3538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:11:33.694214 3538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:11:33.694386 3538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:11:33.702180 3538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:11:33.702310 3538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:11:33.702518 3538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:11:33.712885 3538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:38029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=39463
--tserver_master_addrs=127.18.159.190:44923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:11:33.714247 3538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:11:33.715656 3538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:11:33.730952 3546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:33.730983 3544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:33.731031 3543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:11:33.731865 3538 server_base.cc:1034] running on GCE node
I20260926 02:11:33.732577 3538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:11:33.733671 3538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:11:33.734913 3538 hybrid_clock.cc:648] HybridClock initialized: now 1790388693734866 us; error 38 us; skew 500 ppm
I20260926 02:11:33.735293 3538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:11:33 dist-test-slave-03rn krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388693, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:11:33.738696 3538 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:11:33.740437 3538 webserver.cc:460] Webserver started at http://127.18.159.131:39463/ using document root <none> and password file <none>
I20260926 02:11:33.741201 3538 fs_manager.cc:359] Metadata directory not provided
I20260926 02:11:33.741359 3538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:11:33.747274 3538 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 02:11:33.750521 3553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:33.752226 3538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:11:33.752421 3538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "8ebf3a462272420e821e8b9e2f8798ae"
format_stamp: "Formatted at 2026-09-26 02:10:47 on dist-test-slave-03rn"
server_key: "978cacf6acdcc7c73fae1fcb68955816"
server_key_iv: "8ed615e48ee58f20ac8b64f712bee89e"
server_key_version: "encryptionkey@0"
I20260926 02:11:33.753465 3538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:11:33.766857 3538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:11:33.771934 3538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:11:33.772393 3538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:11:33.773614 3538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:11:33.774989 3538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:11:33.775079 3538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:33.775151 3538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:11:33.775205 3538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:33.810318 3538 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:38029
I20260926 02:11:33.810416 3666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:38029 every 8 connection(s)
I20260926 02:11:33.811882 3538 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:11:33.812727 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 3538
I20260926 02:11:33.817128 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:33.815167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.131:60581)
0926 02:11:33.815908 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:33.816225 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":301,"threads_started":1}
W20260926 02:11:33.817885 3667 heartbeater.cc:644] Failed to heartbeat to 127.18.159.190:44923 (0 consecutive failures): Network error: Failed to ping master at 127.18.159.190:44923: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
I20260926 02:11:33.818852 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:33.818448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.131:37249)
0926 02:11:33.818607 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:33.818708 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 02:11:33.819777 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:33.819470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.131:58549)
0926 02:11:33.819599 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:33.819671 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 02:11:33.820050 3667 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 02:11:33.874261 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:33.873418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:42481)
0926 02:11:33.873964 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:33.874058 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":304,"threads_started":1}
I20260926 02:11:34.326323 3672 raft_consensus.cc:482] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:11:34.326671 3672 raft_consensus.cc:504] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } }
I20260926 02:11:34.328321 3672 leader_election.cc:288] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ebf3a462272420e821e8b9e2f8798ae (127.18.159.131:38029), 4544ec3626644f498d99b4f9408d4eae (127.18.159.130:34015)
Sep 26 02:11:34 dist-test-slave-03rn krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388692, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.131@KRBTEST.COM
Sep 26 02:11:34 dist-test-slave-03rn krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388692, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.130@KRBTEST.COM
I20260926 02:11:34.347450 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:34.345905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.130:51119)
0926 02:11:34.346811 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:34.347054 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":507,"threads_started":1}
I20260926 02:11:34.353602 3674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:34.328900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:60863 (local address 127.18.159.131:38029)
0926 02:11:34.330899 (+ 1999us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:34.330915 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:34.331292 (+ 377us) server_negotiation.cc:407] Connection header received
0926 02:11:34.332899 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:34.332908 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:34.333330 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:34.333681 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:34.335663 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:34.337630 (+ 1967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:34.339515 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:34.340266 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:34.340482 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:34.345635 (+ 5153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:34.345648 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:34.345670 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:34.345731 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:34.349538 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:34.350487 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:34.350499 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:34.350505 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:34.350677 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:34.351179 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:34.351186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:34.351189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:34.351726 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:34.351926 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:34.352790 (+ 864us) server_negotiation.cc:299] Negotiation successful
0926 02:11:34.353162 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1819,"thread_start_us":1712,"threads_started":1}
I20260926 02:11:34.353613 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:34.328625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:34015 (local address 127.18.159.129:34361)
0926 02:11:34.328958 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:34.329016 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:34.329713 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:34.333445 (+ 3732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:34.333456 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:34.334383 (+ 927us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:11:34.335403 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:34.335435 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:34.337200 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:34.337204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:34.339046 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:34.339070 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:34.340634 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:34.340640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:34.340909 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:34.341825 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:11:34.341849 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:11:34.345459 (+ 3610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:11:34.349681 (+ 4222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:11:34.349691 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:11:34.349714 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:11:34.350329 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:11:34.350809 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:11:34.350815 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:11:34.350818 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:11:34.351037 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:11:34.351879 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:11:34.351887 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:11:34.352283 (+ 396us) client_negotiation.cc:769] Sending connection context
0926 02:11:34.352576 (+ 293us) client_negotiation.cc:240] Negotiation successful
0926 02:11:34.353145 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":109,"spinlock_wait_cycles":57216}
I20260926 02:11:34.353613 3675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:34.328901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:34361 (local address 127.18.159.130:34015)
0926 02:11:34.330629 (+ 1728us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:34.330652 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:34.330703 (+ 51us) server_negotiation.cc:407] Connection header received
0926 02:11:34.332491 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:34.332501 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:34.332918 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:34.333225 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:34.335628 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:34.336990 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:34.339246 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:34.340387 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:34.340909 (+ 522us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:34.345803 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:34.345815 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:34.345834 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:34.345881 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:34.349492 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:34.350478 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:34.350488 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:34.350503 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:34.350683 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:34.351151 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:34.351157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:34.351160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:34.351726 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:34.351932 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:34.352802 (+ 870us) server_negotiation.cc:299] Negotiation successful
0926 02:11:34.353196 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1487,"thread_start_us":1332,"threads_started":1}
I20260926 02:11:34.353758 3673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:34.328660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:38029 (local address 127.18.159.129:60863)
0926 02:11:34.331156 (+ 2496us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:34.331175 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:34.331342 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:34.333667 (+ 2325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:34.333672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:34.334389 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:11:34.335446 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:34.335462 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:34.337823 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:34.337829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:34.339352 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:34.339364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:34.340467 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:34.340474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:34.340763 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:34.341810 (+ 1047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:11:34.341843 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:11:34.345206 (+ 3363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:11:34.349714 (+ 4508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:11:34.349719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:11:34.349723 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:11:34.350323 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:11:34.350811 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:11:34.350817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:11:34.350821 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:11:34.351067 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:11:34.351879 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:11:34.351886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:11:34.352283 (+ 397us) client_negotiation.cc:769] Sending connection context
0926 02:11:34.352576 (+ 293us) client_negotiation.cc:240] Negotiation successful
0926 02:11:34.353314 (+ 738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2392,"mutex_wait_us":375,"spinlock_wait_cycles":11008,"thread_start_us":2256,"threads_started":1}
W20260926 02:11:34.357096 3288 leader_election.cc:341] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4544ec3626644f498d99b4f9408d4eae (127.18.159.130:34015): Not found: Tablet not found: 7857dc6a10964493ab3c9f34ba851936
W20260926 02:11:34.357524 3287 leader_election.cc:341] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ebf3a462272420e821e8b9e2f8798ae (127.18.159.131:38029): Not found: Tablet not found: 7857dc6a10964493ab3c9f34ba851936
I20260926 02:11:34.357692 3287 leader_election.cc:302] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79231b61c8954168950f91e8d03dd1f3; no voters: 4544ec3626644f498d99b4f9408d4eae, 8ebf3a462272420e821e8b9e2f8798ae
I20260926 02:11:34.358186 3672 raft_consensus.cc:2726] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 02:11:34.821765 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:34.820771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.131:59391)
0926 02:11:34.821244 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:34.821472 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":220,"threads_started":1}
I20260926 02:11:34.875845 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:34.875120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:39363)
0926 02:11:34.875585 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:34.875675 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":240,"threads_started":1}
I20260926 02:11:35.349680 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:35.348985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.130:42761)
0926 02:11:35.349436 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:35.349523 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":234,"threads_started":1}
I20260926 02:11:35.823961 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:35.822845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.131:46587)
0926 02:11:35.823621 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:35.823752 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":458,"threads_started":1}
I20260926 02:11:35.877858 3681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:35.876990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:45429)
0926 02:11:35.877556 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:35.877649 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":257,"threads_started":1}
I20260926 02:11:36.130228 3682 raft_consensus.cc:482] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:11:36.130472 3682 raft_consensus.cc:504] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } }
I20260926 02:11:36.131325 3682 leader_election.cc:288] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ebf3a462272420e821e8b9e2f8798ae (127.18.159.131:38029), 4544ec3626644f498d99b4f9408d4eae (127.18.159.130:34015)
W20260926 02:11:36.132470 3287 leader_election.cc:341] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ebf3a462272420e821e8b9e2f8798ae (127.18.159.131:38029): Not found: Tablet not found: 7857dc6a10964493ab3c9f34ba851936
W20260926 02:11:36.132692 3288 leader_election.cc:341] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4544ec3626644f498d99b4f9408d4eae (127.18.159.130:34015): Not found: Tablet not found: 7857dc6a10964493ab3c9f34ba851936
I20260926 02:11:36.132869 3288 leader_election.cc:302] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79231b61c8954168950f91e8d03dd1f3; no voters: 4544ec3626644f498d99b4f9408d4eae, 8ebf3a462272420e821e8b9e2f8798ae
I20260926 02:11:36.133158 3682 raft_consensus.cc:2726] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 02:11:36.351691 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:36.350751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.130:49025)
0926 02:11:36.351318 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:36.351450 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":314,"threads_started":1}
I20260926 02:11:36.825923 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:36.825133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.131:49087)
0926 02:11:36.825657 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:36.825750 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":288,"threads_started":1}
I20260926 02:11:36.879628 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:36.878890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:35947)
0926 02:11:36.879376 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:36.879466 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":241,"threads_started":1}
I20260926 02:11:37.353585 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:37.352827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.130:55465)
0926 02:11:37.353310 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:37.353399 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":252,"threads_started":1}
I20260926 02:11:37.827711 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:37.826968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.131:59191)
0926 02:11:37.827396 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:37.827507 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":207,"threads_started":1}
I20260926 02:11:37.881625 3688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:37.880852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:51497)
0926 02:11:37.881301 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:37.881381 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":241,"threads_started":1}
I20260926 02:11:38.355428 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:38.354631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.130:52843)
0926 02:11:38.355136 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:38.355226 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":272,"threads_started":1}
I20260926 02:11:38.710911 3690 raft_consensus.cc:482] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:11:38.711158 3690 raft_consensus.cc:504] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } }
I20260926 02:11:38.711922 3690 leader_election.cc:288] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ebf3a462272420e821e8b9e2f8798ae (127.18.159.131:38029), 4544ec3626644f498d99b4f9408d4eae (127.18.159.130:34015)
W20260926 02:11:38.712989 3288 leader_election.cc:341] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4544ec3626644f498d99b4f9408d4eae (127.18.159.130:34015): Not found: Tablet not found: 7857dc6a10964493ab3c9f34ba851936
W20260926 02:11:38.713198 3287 leader_election.cc:341] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ebf3a462272420e821e8b9e2f8798ae (127.18.159.131:38029): Not found: Tablet not found: 7857dc6a10964493ab3c9f34ba851936
I20260926 02:11:38.713378 3287 leader_election.cc:302] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79231b61c8954168950f91e8d03dd1f3; no voters: 4544ec3626644f498d99b4f9408d4eae, 8ebf3a462272420e821e8b9e2f8798ae
I20260926 02:11:38.713691 3690 raft_consensus.cc:2726] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 02:11:38.829578 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:38.828790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.131:32889)
0926 02:11:38.829283 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:38.829372 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":260,"threads_started":1}
I20260926 02:11:38.883502 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:38.882656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:43643)
0926 02:11:38.883164 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:38.883280 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":242,"threads_started":1}
I20260926 02:11:39.357388 3693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:39.356580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.130:48135)
0926 02:11:39.357127 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:39.357217 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":321,"threads_started":1}
I20260926 02:11:39.831537 3694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:39.830582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.131:52651)
0926 02:11:39.831149 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:39.831291 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":326,"threads_started":1}
I20260926 02:11:39.885574 3695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:39.884681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:54515)
0926 02:11:39.885214 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:39.885321 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":257,"threads_started":1}
I20260926 02:11:40.359318 3696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:40.358397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.130:45751)
0926 02:11:40.359014 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:40.359108 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":258,"threads_started":1}
I20260926 02:11:40.833415 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:40.832658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.131:40103)
0926 02:11:40.833144 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:40.833232 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":220,"threads_started":1}
I20260926 02:11:40.887403 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:40.886560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:50793)
0926 02:11:40.887125 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:40.887243 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":312,"threads_started":1}
I20260926 02:11:41.361068 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:41.360344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.130:41545)
0926 02:11:41.360789 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:41.360884 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":221,"threads_started":1}
I20260926 02:11:41.835070 3700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:41.834351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.131:37097)
0926 02:11:41.834796 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:41.834881 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":247,"threads_started":1}
I20260926 02:11:41.889375 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:41.888501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:49635)
0926 02:11:41.889036 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:41.889161 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":261,"threads_started":1}
I20260926 02:11:42.362990 3702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:42.362139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.130:54265)
0926 02:11:42.362682 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:42.362779 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":262,"threads_started":1}
I20260926 02:11:42.397228 3703 raft_consensus.cc:482] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:11:42.397444 3703 raft_consensus.cc:504] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } }
I20260926 02:11:42.398221 3703 leader_election.cc:288] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ebf3a462272420e821e8b9e2f8798ae (127.18.159.131:38029), 4544ec3626644f498d99b4f9408d4eae (127.18.159.130:34015)
W20260926 02:11:42.399153 3287 leader_election.cc:341] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ebf3a462272420e821e8b9e2f8798ae (127.18.159.131:38029): Not found: Tablet not found: 7857dc6a10964493ab3c9f34ba851936
W20260926 02:11:42.399369 3288 leader_election.cc:341] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4544ec3626644f498d99b4f9408d4eae (127.18.159.130:34015): Not found: Tablet not found: 7857dc6a10964493ab3c9f34ba851936
I20260926 02:11:42.399516 3288 leader_election.cc:302] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79231b61c8954168950f91e8d03dd1f3; no voters: 4544ec3626644f498d99b4f9408d4eae, 8ebf3a462272420e821e8b9e2f8798ae
I20260926 02:11:42.399835 3703 raft_consensus.cc:2726] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 02:11:42.837041 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:42.836266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.131:48457)
0926 02:11:42.836774 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:42.836883 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":257,"threads_started":1}
I20260926 02:11:42.891345 3705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:42.890523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:52705)
0926 02:11:42.891023 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:42.891136 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":247,"threads_started":1}
I20260926 02:11:43.364710 3706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:43.363921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.130:43873)
0926 02:11:43.364441 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:43.364531 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":308,"threads_started":1}
I20260926 02:11:43.838727 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:43.838039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.131:41571)
0926 02:11:43.838470 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:43.838552 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":232,"threads_started":1}
I20260926 02:11:43.893507 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:43.892676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:36735)
0926 02:11:43.893208 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:43.893316 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":272,"threads_started":1}
I20260926 02:11:44.366860 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:44.365855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.130:58051)
0926 02:11:44.366501 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:44.366617 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":301,"threads_started":1}
I20260926 02:11:44.840726 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:44.839785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.131:47897)
0926 02:11:44.840466 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:44.840568 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":415,"threads_started":1}
I20260926 02:11:44.895413 3711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:44.894647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:55361)
0926 02:11:44.895155 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:44.895249 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":253,"threads_started":1}
I20260926 02:11:45.368721 3712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:45.367897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.130:35379)
0926 02:11:45.368441 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:45.368529 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":300,"threads_started":1}
I20260926 02:11:45.842427 3713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:45.841716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.131:35617)
0926 02:11:45.842161 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:45.842247 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":228,"threads_started":1}
I20260926 02:11:45.897348 3714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:45.896494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:51335)
0926 02:11:45.897065 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:45.897148 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":315,"threads_started":1}
I20260926 02:11:46.370720 3715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:46.369797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.130:43693)
0926 02:11:46.370340 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:46.370486 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":251,"threads_started":1}
I20260926 02:11:46.840576 3716 raft_consensus.cc:482] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:11:46.840802 3716 raft_consensus.cc:504] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebf3a462272420e821e8b9e2f8798ae" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 38029 } } peers { permanent_uuid: "4544ec3626644f498d99b4f9408d4eae" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 34015 } } peers { permanent_uuid: "79231b61c8954168950f91e8d03dd1f3" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 43893 } }
I20260926 02:11:46.841509 3716 leader_election.cc:288] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ebf3a462272420e821e8b9e2f8798ae (127.18.159.131:38029), 4544ec3626644f498d99b4f9408d4eae (127.18.159.130:34015)
W20260926 02:11:46.842654 3288 leader_election.cc:341] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4544ec3626644f498d99b4f9408d4eae (127.18.159.130:34015): Not found: Tablet not found: 7857dc6a10964493ab3c9f34ba851936
W20260926 02:11:46.842896 3287 leader_election.cc:341] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ebf3a462272420e821e8b9e2f8798ae (127.18.159.131:38029): Not found: Tablet not found: 7857dc6a10964493ab3c9f34ba851936
I20260926 02:11:46.843026 3287 leader_election.cc:302] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79231b61c8954168950f91e8d03dd1f3; no voters: 4544ec3626644f498d99b4f9408d4eae, 8ebf3a462272420e821e8b9e2f8798ae
I20260926 02:11:46.843324 3716 raft_consensus.cc:2726] T 7857dc6a10964493ab3c9f34ba851936 P 79231b61c8954168950f91e8d03dd1f3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 02:11:46.844215 3717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:46.843508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.131:54357)
0926 02:11:46.843930 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:46.844031 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":199,"threads_started":1}
I20260926 02:11:46.899228 3718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:46.898292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:55443)
0926 02:11:46.898876 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:46.899004 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":298,"threads_started":1}
I20260926 02:11:47.372615 3719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:47.371687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.130:42501)
0926 02:11:47.372270 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:47.372372 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":293,"threads_started":1}
I20260926 02:11:47.846143 3720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:47.845179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.131:52867)
0926 02:11:47.845760 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:47.845860 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":337,"threads_started":1}
I20260926 02:11:47.901403 3721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:47.900652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:49429)
0926 02:11:47.901132 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:47.901218 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":258,"threads_started":1}
I20260926 02:11:48.375900 3722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:48.374063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.130:55003)
0926 02:11:48.375559 (+ 1496us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:48.375738 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1338,"thread_start_us":1164,"threads_started":1}
I20260926 02:11:48.823525 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 3230
I20260926 02:11:48.839861 3723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:48.824787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.18.159.129:43893)
0926 02:11:48.825359 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:48.825370 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:48.825405 (+ 35us) server_negotiation.cc:407] Connection header received
0926 02:11:48.825489 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:48.825495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:48.825750 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:48.825926 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:48.827615 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:48.828792 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:48.830117 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:48.830679 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:48.830839 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:48.833710 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:48.833723 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:48.833738 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:48.833794 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:48.836825 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:48.837624 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:48.837633 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:48.837635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:48.837776 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:48.838211 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:48.838216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:48.838219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:48.838599 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:48.838921 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:48.839239 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 02:11:48.839535 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":247,"threads_started":1}
I20260926 02:11:48.841305 3396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:11:48.848119 3725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:48.847298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.131:60629)
0926 02:11:48.847815 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:48.847960 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":281,"threads_started":1}
I20260926 02:11:48.932922 3726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:48.931710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:44923 (local address 127.18.159.129:39303)
0926 02:11:48.932349 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:48.932600 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.190:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":307,"threads_started":1}
I20260926 02:11:48.932987 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 3230
I20260926 02:11:48.951735 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 3405
I20260926 02:11:48.966441 3727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:48.952608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47512 (local address 127.18.159.130:34015)
0926 02:11:48.953094 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:48.953099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:48.953114 (+ 15us) server_negotiation.cc:407] Connection header received
0926 02:11:48.953167 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:48.953172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:48.953236 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:48.953356 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:48.954882 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:48.955651 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:48.957113 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:48.957570 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:48.957731 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:48.960654 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:48.960660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:48.960664 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:48.960705 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:48.963381 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:48.964227 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:48.964233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:48.964241 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:48.964402 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:48.964869 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:48.964874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:48.964878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:48.965187 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:48.965506 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:48.965798 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 02:11:48.966050 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":230,"threads_started":1}
I20260926 02:11:48.967167 3530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:11:49.051409 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 3405
I20260926 02:11:49.069415 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 3538
I20260926 02:11:49.083880 3729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:49.070398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48138 (local address 127.18.159.131:38029)
0926 02:11:49.070981 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:49.070986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:49.071004 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:11:49.071061 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:49.071064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:49.071135 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:49.071232 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:49.072812 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:49.073651 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:49.075021 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:49.075438 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:49.075536 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:49.078202 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:49.078207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:49.078211 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:49.078250 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:49.081081 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:49.081819 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:49.081823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:49.081825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:49.082001 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:49.082396 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:49.082402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:49.082405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:49.082714 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:49.082982 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:49.083267 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 02:11:49.083451 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":324,"threads_started":1}
I20260926 02:11:49.084664 3662 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:11:49.170939 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 3538
2026-09-26T02:11:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65097 ms)
[----------] 30 tests from SecurityITest (306965 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:11:49 dist-test-slave-03rn krb5kdc[3733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:11:49 dist-test-slave-03rn krb5kdc[3733](info): set up 2 sockets
Sep 26 02:11:49 dist-test-slave-03rn krb5kdc[3733](info): commencing operation
krb5kdc: starting...
W20260926 02:11:50.293167 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:11:50 dist-test-slave-03rn krb5kdc[3733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:11:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:11:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:11:50.562031 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:35127
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:39229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:35127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:11:50.909137 3747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:11:50.909458 3747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:11:50.909552 3747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:11:50.917119 3747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:11:50.917243 3747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:11:50.917292 3747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:11:50.917344 3747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:11:50.917395 3747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:11:50.927711 3747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:39229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:35127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:35127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:11:50.929111 3747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:11:50.930382 3747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:11:50.943168 3755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:50.943220 3753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:50.943363 3752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:11:50.943948 3747 server_base.cc:1034] running on GCE node
I20260926 02:11:50.944844 3747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:11:50.946409 3747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:11:50.947638 3747 hybrid_clock.cc:648] HybridClock initialized: now 1790388710947585 us; error 41 us; skew 500 ppm
I20260926 02:11:50.948053 3747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:11:50 dist-test-slave-03rn krb5kdc[3733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388710, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:11:50.951525 3747 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:11:50.953301 3747 webserver.cc:460] Webserver started at http://127.18.159.190:44099/ using document root <none> and password file <none>
I20260926 02:11:50.954071 3747 fs_manager.cc:359] Metadata directory not provided
I20260926 02:11:50.954223 3747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:11:50.954553 3747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:11:50.958344 3747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "c63ce4710ee84751a2e26f05f5976508"
format_stamp: "Formatted at 2026-09-26 02:11:50 on dist-test-slave-03rn"
server_key: "3737d7a835056c9d685909fc6ca5dde0"
server_key_iv: "1e5986735864e18929fae4ebb49a7a87"
server_key_version: "encryptionkey@0"
I20260926 02:11:50.959219 3747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "c63ce4710ee84751a2e26f05f5976508"
format_stamp: "Formatted at 2026-09-26 02:11:50 on dist-test-slave-03rn"
server_key: "3737d7a835056c9d685909fc6ca5dde0"
server_key_iv: "1e5986735864e18929fae4ebb49a7a87"
server_key_version: "encryptionkey@0"
I20260926 02:11:50.965673 3747 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 02:11:50.969758 3762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:50.971319 3747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20260926 02:11:50.971550 3747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "c63ce4710ee84751a2e26f05f5976508"
format_stamp: "Formatted at 2026-09-26 02:11:50 on dist-test-slave-03rn"
server_key: "3737d7a835056c9d685909fc6ca5dde0"
server_key_iv: "1e5986735864e18929fae4ebb49a7a87"
server_key_version: "encryptionkey@0"
I20260926 02:11:50.971784 3747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:11:50.985355 3747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:11:50.990151 3747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:11:50.990473 3747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:11:51.010492 3747 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:35127
I20260926 02:11:51.010492 3813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:35127 every 8 connection(s)
I20260926 02:11:51.011983 3747 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:11:51.015897 3814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:11:51.017871 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 3747
I20260926 02:11:51.018080 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:11:51.018894 19070 external_mini_cluster.cc:1428] Setting key 1d1dfd821f2f46b7427323d6468ff7ca
I20260926 02:11:51.029901 3814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508: Bootstrap starting.
Sep 26 02:11:51 dist-test-slave-03rn krb5kdc[3733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:11:51.033339 3814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508: Neither blocks nor log segments found. Creating new log.
I20260926 02:11:51.034453 3814 log.cc:824] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508: Log is configured to *not* fsync() on all Append() calls
I20260926 02:11:51.037891 3814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508: No bootstrap required, opened a new log
I20260926 02:11:51.042271 3817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:51.021193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54836 (local address 127.18.159.190:35127)
0926 02:11:51.022048 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:51.022060 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:51.022106 (+ 46us) server_negotiation.cc:407] Connection header received
0926 02:11:51.023741 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:51.023771 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:51.024287 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:51.024697 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:51.026232 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:51.027633 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:51.029089 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:51.029800 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:51.030022 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:51.034311 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:51.034320 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:51.034336 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:51.034379 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:51.037907 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:51.038705 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:51.038715 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:51.038722 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:51.038883 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:51.039339 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:51.039343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:51.039345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:51.039912 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:51.040275 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:51.040769 (+ 494us) server_negotiation.cc:299] Negotiation successful
0926 02:11:51.041208 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":390,"threads_started":1}
I20260926 02:11:51.046694 3814 raft_consensus.cc:348] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63ce4710ee84751a2e26f05f5976508" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35127 } }
I20260926 02:11:51.047112 3814 raft_consensus.cc:374] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:11:51.047250 3814 raft_consensus.cc:729] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63ce4710ee84751a2e26f05f5976508, State: Initialized, Role: FOLLOWER
I20260926 02:11:51.047890 3814 consensus_queue.cc:260] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63ce4710ee84751a2e26f05f5976508" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35127 } }
I20260926 02:11:51.048148 3814 raft_consensus.cc:388] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:11:51.048321 3814 raft_consensus.cc:482] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:11:51.048501 3814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:11:51.050699 3814 raft_consensus.cc:504] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63ce4710ee84751a2e26f05f5976508" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35127 } }
I20260926 02:11:51.051231 3814 leader_election.cc:302] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c63ce4710ee84751a2e26f05f5976508; no voters:
I20260926 02:11:51.051828 3814 leader_election.cc:288] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:11:51.052038 3819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:11:51.053278 3819 raft_consensus.cc:686] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508 [term 1 LEADER]: Becoming Leader. State: Replica: c63ce4710ee84751a2e26f05f5976508, State: Running, Role: LEADER
I20260926 02:11:51.053840 3819 consensus_queue.cc:237] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63ce4710ee84751a2e26f05f5976508" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35127 } }
I20260926 02:11:51.054207 3814 sys_catalog.cc:564] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:11:51.056833 3821 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c63ce4710ee84751a2e26f05f5976508. Latest consensus state: current_term: 1 leader_uuid: "c63ce4710ee84751a2e26f05f5976508" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63ce4710ee84751a2e26f05f5976508" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35127 } } }
I20260926 02:11:51.056742 3820 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c63ce4710ee84751a2e26f05f5976508" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63ce4710ee84751a2e26f05f5976508" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35127 } } }
I20260926 02:11:51.057253 3821 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508 [sys.catalog]: This master's current role is: LEADER
I20260926 02:11:51.057327 3820 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508 [sys.catalog]: This master's current role is: LEADER
I20260926 02:11:51.059319 3828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:11:51.065248 3828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:11:51.074545 3828 catalog_manager.cc:1347] Generated new cluster ID: ff9e60490f634d7b953372a8f35db58d
I20260926 02:11:51.074693 3828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:11:51.107793 3828 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:11:51.109642 3828 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:11:51.118498 3828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508: Generated new TSK 0
I20260926 02:11:51.119405 3828 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.159.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
I20260926 02:11:51.192528 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35127
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:39229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:11:51.558516 3840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:11:51.558846 3840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:11:51.558992 3840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:11:51.566562 3840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:11:51.566685 3840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:11:51.566802 3840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:11:51.577081 3840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:39229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:11:51.578327 3840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:11:51.579655 3840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:11:51.593621 3845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:51.593743 3848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:51.593720 3846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:11:51.594780 3840 server_base.cc:1034] running on GCE node
I20260926 02:11:51.595485 3840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:11:51.596582 3840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:11:51.597870 3840 hybrid_clock.cc:648] HybridClock initialized: now 1790388711597805 us; error 52 us; skew 500 ppm
I20260926 02:11:51.598268 3840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:11:51 dist-test-slave-03rn krb5kdc[3733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388711, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:11:51.607726 3840 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:11:51.609483 3840 webserver.cc:460] Webserver started at http://127.18.159.129:43685/ using document root <none> and password file <none>
I20260926 02:11:51.610239 3840 fs_manager.cc:359] Metadata directory not provided
I20260926 02:11:51.610391 3840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:11:51.610742 3840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:11:51.614113 3840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "9197eef629df49d3a2f9e9c698b2efe1"
format_stamp: "Formatted at 2026-09-26 02:11:51 on dist-test-slave-03rn"
server_key: "412b46abd5bcccce30a0cda0da13ed6e"
server_key_iv: "26573a66785538bf2a00409872812bd0"
server_key_version: "encryptionkey@0"
I20260926 02:11:51.615015 3840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "9197eef629df49d3a2f9e9c698b2efe1"
format_stamp: "Formatted at 2026-09-26 02:11:51 on dist-test-slave-03rn"
server_key: "412b46abd5bcccce30a0cda0da13ed6e"
server_key_iv: "26573a66785538bf2a00409872812bd0"
server_key_version: "encryptionkey@0"
I20260926 02:11:51.621102 3840 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 02:11:51.624866 3855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:51.626346 3840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:11:51.626574 3840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "9197eef629df49d3a2f9e9c698b2efe1"
format_stamp: "Formatted at 2026-09-26 02:11:51 on dist-test-slave-03rn"
server_key: "412b46abd5bcccce30a0cda0da13ed6e"
server_key_iv: "26573a66785538bf2a00409872812bd0"
server_key_version: "encryptionkey@0"
I20260926 02:11:51.626811 3840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:11:51.643019 3840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:11:51.647719 3840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:11:51.648048 3840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:11:51.649116 3840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:11:51.650875 3840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:11:51.650998 3840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:51.651157 3840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:11:51.651235 3840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:51.686313 3840 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:45147
I20260926 02:11:51.686359 3968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:45147 every 8 connection(s)
I20260926 02:11:51.687726 3840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:11:51.692694 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 3840
I20260926 02:11:51.692914 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:11:51.693559 19070 external_mini_cluster.cc:1428] Setting key 6b016c81ff96e6e41a8ae78af039c744
Sep 26 02:11:51 dist-test-slave-03rn krb5kdc[3733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388711, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:11:51.708602 3971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:51.690317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:40033 (local address 127.18.159.190:35127)
0926 02:11:51.690774 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:51.690778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:51.691622 (+ 844us) server_negotiation.cc:407] Connection header received
0926 02:11:51.693696 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:51.693702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:51.693801 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:51.693928 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:51.695739 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:51.696548 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:51.698180 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:51.698599 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:51.698742 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:51.702454 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:51.702460 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:51.702463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:51.702505 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:51.705237 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:51.706050 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:51.706058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:51.706067 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:51.706196 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:51.706677 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:51.706683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:51.706693 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:51.706984 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:51.707144 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:51.707945 (+ 801us) server_negotiation.cc:299] Negotiation successful
0926 02:11:51.708227 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":254,"threads_started":1}
I20260926 02:11:51.709662 3972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:51.690749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35127 (local address 127.18.159.129:40033)
0926 02:11:51.691448 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:51.691502 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:51.693391 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:51.694190 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:51.694200 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:51.694836 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:11:51.695569 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:51.695588 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:51.696716 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:51.696720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:51.698028 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:51.698041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:51.698845 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:51.698853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:51.699121 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:51.699864 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:11:51.699888 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:11:51.702188 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:11:51.705404 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:11:51.705414 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:11:51.705434 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:11:51.705904 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:11:51.706317 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:11:51.706322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:11:51.706326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:11:51.706544 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:11:51.707118 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:11:51.707128 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:11:51.707553 (+ 425us) client_negotiation.cc:769] Sending connection context
0926 02:11:51.707848 (+ 295us) client_negotiation.cc:240] Negotiation successful
0926 02:11:51.708496 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":276,"threads_started":1}
I20260926 02:11:51.711371 3969 heartbeater.cc:347] Connected to a master server at 127.18.159.190:35127
I20260926 02:11:51.711709 3969 heartbeater.cc:464] Registering TS with master...
I20260926 02:11:51.712566 3969 heartbeater.cc:511] Master 127.18.159.190:35127 requested a full tablet report, sending...
I20260926 02:11:51.714751 3779 ts_manager.cc:194] Registered new tserver with Master: 9197eef629df49d3a2f9e9c698b2efe1 (127.18.159.129:45147)
I20260926 02:11:51.716298 3779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:40033
WARNING: no policy specified for kudu/127.18.159.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
I20260926 02:11:51.751709 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35127
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:39229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:11:52.100080 3975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:11:52.100492 3975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:11:52.100668 3975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:11:52.108450 3975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:11:52.108572 3975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:11:52.108790 3975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:11:52.119099 3975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:39229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:11:52.120512 3975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:11:52.121816 3975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:11:52.135502 3983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:52.135591 3981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:52.135843 3980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:11:52.136660 3975 server_base.cc:1034] running on GCE node
I20260926 02:11:52.137359 3975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:11:52.138468 3975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:11:52.139729 3975 hybrid_clock.cc:648] HybridClock initialized: now 1790388712139669 us; error 57 us; skew 500 ppm
I20260926 02:11:52.140101 3975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:11:52 dist-test-slave-03rn krb5kdc[3733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388712, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:11:52.143515 3975 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:11:52.145160 3975 webserver.cc:460] Webserver started at http://127.18.159.130:43951/ using document root <none> and password file <none>
I20260926 02:11:52.145880 3975 fs_manager.cc:359] Metadata directory not provided
I20260926 02:11:52.146034 3975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:11:52.146353 3975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:11:52.149390 3975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "23eb7ca300bd4112926c8d81ed3292e1"
format_stamp: "Formatted at 2026-09-26 02:11:52 on dist-test-slave-03rn"
server_key: "ef368a6de18941a08151e21108ddb74c"
server_key_iv: "5512d8651ee20e158d451f3b12447be2"
server_key_version: "encryptionkey@0"
I20260926 02:11:52.150164 3975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "23eb7ca300bd4112926c8d81ed3292e1"
format_stamp: "Formatted at 2026-09-26 02:11:52 on dist-test-slave-03rn"
server_key: "ef368a6de18941a08151e21108ddb74c"
server_key_iv: "5512d8651ee20e158d451f3b12447be2"
server_key_version: "encryptionkey@0"
I20260926 02:11:52.155642 3975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:11:52.159447 3990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:52.160916 3975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:11:52.161092 3975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "23eb7ca300bd4112926c8d81ed3292e1"
format_stamp: "Formatted at 2026-09-26 02:11:52 on dist-test-slave-03rn"
server_key: "ef368a6de18941a08151e21108ddb74c"
server_key_iv: "5512d8651ee20e158d451f3b12447be2"
server_key_version: "encryptionkey@0"
I20260926 02:11:52.161312 3975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:11:52.197638 3975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:11:52.202133 3975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:11:52.202443 3975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:11:52.203413 3975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:11:52.204813 3975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:11:52.204914 3975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:52.205067 3975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:11:52.205139 3975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:52.239202 3975 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:39241
I20260926 02:11:52.239321 4103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:39241 every 8 connection(s)
I20260926 02:11:52.240610 3975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:11:52.247850 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 3975
I20260926 02:11:52.248013 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:11:52.248559 19070 external_mini_cluster.cc:1428] Setting key c51ca047cba36b8aab7bc83b22f79d66
Sep 26 02:11:52 dist-test-slave-03rn krb5kdc[3733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388712, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:11:52.264868 4106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:52.243279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:43903 (local address 127.18.159.190:35127)
0926 02:11:52.243748 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:52.243754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:52.244891 (+ 1137us) server_negotiation.cc:407] Connection header received
0926 02:11:52.246784 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:52.246788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:52.246864 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:52.246971 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:52.249208 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.250153 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.252002 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.252881 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.253041 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:52.257126 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:52.257137 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:52.257144 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:52.257197 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:52.260897 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:52.262041 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:52.262046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:52.262049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:52.262195 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:52.262751 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:52.262755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:52.262757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:52.263057 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:52.263177 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:52.264293 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0926 02:11:52.264536 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":228,"threads_started":1}
I20260926 02:11:52.266359 4107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:52.243903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35127 (local address 127.18.159.130:43903)
0926 02:11:52.244721 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:52.244762 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:52.246474 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:52.247470 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:52.247480 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:52.248256 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:11:52.248960 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:52.248981 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.250394 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.250399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:52.251815 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:52.251830 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.253121 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.253126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:52.253350 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:52.254159 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:11:52.254192 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:11:52.256835 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:11:52.261082 (+ 4247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:11:52.261095 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:11:52.261118 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:11:52.261840 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:11:52.262320 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:11:52.262328 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:11:52.262334 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:11:52.262615 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:11:52.263223 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:11:52.263235 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:11:52.263752 (+ 517us) client_negotiation.cc:769] Sending connection context
0926 02:11:52.264085 (+ 333us) client_negotiation.cc:240] Negotiation successful
0926 02:11:52.264767 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":385,"threads_started":1}
I20260926 02:11:52.268064 4104 heartbeater.cc:347] Connected to a master server at 127.18.159.190:35127
I20260926 02:11:52.268456 4104 heartbeater.cc:464] Registering TS with master...
I20260926 02:11:52.269258 4104 heartbeater.cc:511] Master 127.18.159.190:35127 requested a full tablet report, sending...
I20260926 02:11:52.271246 3779 ts_manager.cc:194] Registered new tserver with Master: 23eb7ca300bd4112926c8d81ed3292e1 (127.18.159.130:39241)
I20260926 02:11:52.272611 3779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:43903
WARNING: no policy specified for kudu/127.18.159.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
I20260926 02:11:52.307235 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35127
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:39229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:11:52.650996 4110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:11:52.651351 4110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:11:52.651522 4110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:11:52.659461 4110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:11:52.659629 4110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:11:52.659819 4110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:11:52.670109 4110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:39229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:35127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:11:52.671505 4110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:11:52.672972 4110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:11:52.689924 4115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:52.690048 4116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:52.690419 4118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:11:52.691152 4110 server_base.cc:1034] running on GCE node
I20260926 02:11:52.691841 4110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:11:52.692925 4110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:11:52.694155 4110 hybrid_clock.cc:648] HybridClock initialized: now 1790388712694110 us; error 42 us; skew 500 ppm
I20260926 02:11:52.694548 4110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:11:52 dist-test-slave-03rn krb5kdc[3733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388712, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:11:52.697872 4110 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:11:52.699460 4110 webserver.cc:460] Webserver started at http://127.18.159.131:33725/ using document root <none> and password file <none>
I20260926 02:11:52.700269 4110 fs_manager.cc:359] Metadata directory not provided
I20260926 02:11:52.700438 4110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:11:52.700775 4110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:11:52.703867 4110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "d215f6ce0d924313809e6ca7bd14b0da"
format_stamp: "Formatted at 2026-09-26 02:11:52 on dist-test-slave-03rn"
server_key: "daa86524c377bf78788ca3d994190179"
server_key_iv: "e75d6afa95da6f05bcceb8748bb2adfa"
server_key_version: "encryptionkey@0"
I20260926 02:11:52.704741 4110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "d215f6ce0d924313809e6ca7bd14b0da"
format_stamp: "Formatted at 2026-09-26 02:11:52 on dist-test-slave-03rn"
server_key: "daa86524c377bf78788ca3d994190179"
server_key_iv: "e75d6afa95da6f05bcceb8748bb2adfa"
server_key_version: "encryptionkey@0"
I20260926 02:11:52.710258 4110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:11:52.714159 4125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:52.715641 4110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:11:52.715837 4110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "d215f6ce0d924313809e6ca7bd14b0da"
format_stamp: "Formatted at 2026-09-26 02:11:52 on dist-test-slave-03rn"
server_key: "daa86524c377bf78788ca3d994190179"
server_key_iv: "e75d6afa95da6f05bcceb8748bb2adfa"
server_key_version: "encryptionkey@0"
I20260926 02:11:52.716058 4110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:11:52.719508 3969 heartbeater.cc:503] Master 127.18.159.190:35127 was elected leader, sending a full tablet report...
I20260926 02:11:52.731839 4110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:11:52.736640 4110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:11:52.736960 4110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:11:52.737957 4110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:11:52.739360 4110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:11:52.739466 4110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:52.739611 4110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:11:52.739687 4110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:52.773216 4110 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:44443
I20260926 02:11:52.773263 4238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:44443 every 8 connection(s)
I20260926 02:11:52.774572 4110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:11:52.783964 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 4110
I20260926 02:11:52.784186 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:11:52.784742 19070 external_mini_cluster.cc:1428] Setting key f0824f0ee95d955252a689f3be332b53
Sep 26 02:11:52 dist-test-slave-03rn krb5kdc[3733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388712, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:11:52.795818 4241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:52.777272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:43321 (local address 127.18.159.190:35127)
0926 02:11:52.777740 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:52.777745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:52.778566 (+ 821us) server_negotiation.cc:407] Connection header received
0926 02:11:52.780484 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:52.780491 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:52.780579 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:52.780699 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:52.782301 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.783107 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.785460 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.785902 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.786044 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:52.789531 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:52.789537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:52.789540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:52.789574 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:52.792639 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:52.793408 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:52.793414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:52.793418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:52.793558 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:52.793998 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:52.794004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:52.794007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:52.794288 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:52.794446 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:52.795189 (+ 743us) server_negotiation.cc:299] Negotiation successful
0926 02:11:52.795418 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":250,"threads_started":1}
I20260926 02:11:52.796844 4242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:52.777718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35127 (local address 127.18.159.131:43321)
0926 02:11:52.778404 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:52.778445 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:52.780135 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:52.780946 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:52.780955 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:52.781528 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:11:52.782118 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:52.782137 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.783279 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.783283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:52.785307 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:52.785320 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.786137 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.786144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:52.786399 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:52.787079 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:11:52.787102 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:11:52.789291 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:11:52.792788 (+ 3497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:11:52.792799 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:11:52.792817 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:11:52.793273 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:11:52.793673 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:11:52.793678 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:11:52.793682 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:11:52.793879 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:11:52.794425 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:11:52.794433 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:11:52.794795 (+ 362us) client_negotiation.cc:769] Sending connection context
0926 02:11:52.795087 (+ 292us) client_negotiation.cc:240] Negotiation successful
0926 02:11:52.795655 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":271,"threads_started":1}
I20260926 02:11:52.798462 4239 heartbeater.cc:347] Connected to a master server at 127.18.159.190:35127
I20260926 02:11:52.798872 4239 heartbeater.cc:464] Registering TS with master...
I20260926 02:11:52.799907 4239 heartbeater.cc:511] Master 127.18.159.190:35127 requested a full tablet report, sending...
I20260926 02:11:52.801609 3779 ts_manager.cc:194] Registered new tserver with Master: d215f6ce0d924313809e6ca7bd14b0da (127.18.159.131:44443)
I20260926 02:11:52.802497 3779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:43321
I20260926 02:11:52.813162 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:11:52.829871 4241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:52.817584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54844 (local address 127.18.159.190:35127)
0926 02:11:52.817759 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:52.817763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:52.818152 (+ 389us) server_negotiation.cc:407] Connection header received
0926 02:11:52.818309 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:52.818312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:52.818389 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:52.818486 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:52.820000 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.820840 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.822166 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.822556 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.822648 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:52.825042 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:52.825048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:52.825051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:52.825085 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:52.827526 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:52.828188 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:52.828192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:52.828194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:52.828306 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:52.828635 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:52.828638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:52.828639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:52.828947 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:52.829173 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:52.829401 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 02:11:52.829583 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:11:52.834043 3779 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:11:52.841142 3779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:11:52.866173 4252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:52.853933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:39241 (local address 127.0.0.1:43666)
0926 02:11:52.854586 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:52.854634 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:52.854812 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:52.855838 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:52.855847 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:52.855879 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:11:52.856315 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:52.856341 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.858807 (+ 2466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.858816 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:52.861912 (+ 3096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:52.861940 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.864748 (+ 2808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.864759 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:52.865096 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:52.865178 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 02:11:52.865351 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 02:11:52.865634 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":281,"threads_started":1}
I20260926 02:11:52.867641 4253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:52.854017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43666 (local address 127.18.159.130:39241)
0926 02:11:52.854615 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:52.854623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:52.854744 (+ 121us) server_negotiation.cc:407] Connection header received
0926 02:11:52.855050 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:52.855062 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:52.855395 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:52.855605 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:11:52.856650 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.858609 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.862181 (+ 3572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.864088 (+ 1907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.864326 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:52.864584 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:52.866901 (+ 2317us) server_negotiation.cc:299] Negotiation successful
0926 02:11:52.867171 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":325,"threads_started":1}
I20260926 02:11:52.874382 4039 tablet_service.cc:1467] Processing CreateTablet for tablet 02ce96cb3b4c426a9951533dabc0ce86 (DEFAULT_TABLE table=test-table [id=ad66be8644e84581934f3d76dd6542ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:11:52.875944 4039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ce96cb3b4c426a9951533dabc0ce86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:11:52.876225 4252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:52.855385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:45147 (local address 127.0.0.1:42780)
0926 02:11:52.866528 (+ 11143us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:52.866554 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:52.866683 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:52.868981 (+ 2298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:52.868986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:52.869012 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:11:52.869220 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:52.869234 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.871190 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.871195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:52.873157 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:52.873172 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.875191 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.875197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:52.875435 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:52.875479 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 02:11:52.875583 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 02:11:52.875746 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11010,"thread_start_us":8257,"threads_started":1}
I20260926 02:11:52.876241 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:52.855942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.18.159.129:45147)
0926 02:11:52.868437 (+ 12495us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:52.868444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:52.868472 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:11:52.868554 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:52.868561 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:52.868772 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:52.869011 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:11:52.869425 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.871025 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.873356 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.874887 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.875091 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:52.875281 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:52.875658 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 02:11:52.875833 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12320,"thread_start_us":12136,"threads_started":1}
I20260926 02:11:52.878279 4255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:52.855036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:44443 (local address 127.0.0.1:39562)
0926 02:11:52.864057 (+ 9021us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:52.864085 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:52.865365 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:52.870685 (+ 5320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:52.870690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:52.870720 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:11:52.871112 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:52.871124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.873177 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.873182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:52.875265 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:52.875280 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.877529 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.877541 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:52.877727 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:52.877761 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 02:11:52.877855 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 02:11:52.878025 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8815,"thread_start_us":14341,"threads_started":1}
I20260926 02:11:52.878932 4256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:52.855508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39562 (local address 127.18.159.131:44443)
0926 02:11:52.870049 (+ 14541us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:52.870058 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:52.870094 (+ 36us) server_negotiation.cc:407] Connection header received
0926 02:11:52.870190 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:52.870198 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:52.870440 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:52.870642 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:11:52.871335 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.873001 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.875452 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:52.876993 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:52.877187 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:52.877370 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:52.878544 (+ 1174us) server_negotiation.cc:299] Negotiation successful
0926 02:11:52.878694 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14356,"thread_start_us":14178,"threads_started":1}
I20260926 02:11:52.880951 3904 tablet_service.cc:1467] Processing CreateTablet for tablet 02ce96cb3b4c426a9951533dabc0ce86 (DEFAULT_TABLE table=test-table [id=ad66be8644e84581934f3d76dd6542ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:11:52.882577 3904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ce96cb3b4c426a9951533dabc0ce86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:11:52.882676 4174 tablet_service.cc:1467] Processing CreateTablet for tablet 02ce96cb3b4c426a9951533dabc0ce86 (DEFAULT_TABLE table=test-table [id=ad66be8644e84581934f3d76dd6542ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:11:52.883939 4174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ce96cb3b4c426a9951533dabc0ce86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:11:52.894534 4258 tablet_bootstrap.cc:492] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1: Bootstrap starting.
I20260926 02:11:52.896824 4259 tablet_bootstrap.cc:492] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da: Bootstrap starting.
I20260926 02:11:52.898725 4258 tablet_bootstrap.cc:654] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1: Neither blocks nor log segments found. Creating new log.
I20260926 02:11:52.900039 4258 log.cc:824] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1: Log is configured to *not* fsync() on all Append() calls
I20260926 02:11:52.902176 4259 tablet_bootstrap.cc:654] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da: Neither blocks nor log segments found. Creating new log.
I20260926 02:11:52.903649 4260 tablet_bootstrap.cc:492] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1: Bootstrap starting.
I20260926 02:11:52.903717 4259 log.cc:824] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da: Log is configured to *not* fsync() on all Append() calls
I20260926 02:11:52.907943 4260 tablet_bootstrap.cc:654] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1: Neither blocks nor log segments found. Creating new log.
I20260926 02:11:52.908635 4259 tablet_bootstrap.cc:492] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da: No bootstrap required, opened a new log
I20260926 02:11:52.908844 4258 tablet_bootstrap.cc:492] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1: No bootstrap required, opened a new log
I20260926 02:11:52.908958 4259 ts_tablet_manager.cc:1397] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20260926 02:11:52.909138 4258 ts_tablet_manager.cc:1397] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260926 02:11:52.909288 4260 log.cc:824] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1: Log is configured to *not* fsync() on all Append() calls
I20260926 02:11:52.913388 4260 tablet_bootstrap.cc:492] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1: No bootstrap required, opened a new log
I20260926 02:11:52.913672 4260 ts_tablet_manager.cc:1397] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 02:11:52.917619 4259 raft_consensus.cc:348] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eb7ca300bd4112926c8d81ed3292e1" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39241 } } peers { permanent_uuid: "d215f6ce0d924313809e6ca7bd14b0da" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 44443 } } peers { permanent_uuid: "9197eef629df49d3a2f9e9c698b2efe1" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 45147 } }
I20260926 02:11:52.918021 4259 raft_consensus.cc:374] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:11:52.918119 4259 raft_consensus.cc:729] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d215f6ce0d924313809e6ca7bd14b0da, State: Initialized, Role: FOLLOWER
I20260926 02:11:52.918803 4259 consensus_queue.cc:260] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eb7ca300bd4112926c8d81ed3292e1" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39241 } } peers { permanent_uuid: "d215f6ce0d924313809e6ca7bd14b0da" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 44443 } } peers { permanent_uuid: "9197eef629df49d3a2f9e9c698b2efe1" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 45147 } }
I20260926 02:11:52.919766 4258 raft_consensus.cc:348] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eb7ca300bd4112926c8d81ed3292e1" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39241 } } peers { permanent_uuid: "d215f6ce0d924313809e6ca7bd14b0da" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 44443 } } peers { permanent_uuid: "9197eef629df49d3a2f9e9c698b2efe1" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 45147 } }
I20260926 02:11:52.920284 4258 raft_consensus.cc:374] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:11:52.920392 4258 raft_consensus.cc:729] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eb7ca300bd4112926c8d81ed3292e1, State: Initialized, Role: FOLLOWER
I20260926 02:11:52.921057 4258 consensus_queue.cc:260] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eb7ca300bd4112926c8d81ed3292e1" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39241 } } peers { permanent_uuid: "d215f6ce0d924313809e6ca7bd14b0da" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 44443 } } peers { permanent_uuid: "9197eef629df49d3a2f9e9c698b2efe1" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 45147 } }
I20260926 02:11:52.925068 4239 heartbeater.cc:503] Master 127.18.159.190:35127 was elected leader, sending a full tablet report...
I20260926 02:11:52.925527 4104 heartbeater.cc:503] Master 127.18.159.190:35127 was elected leader, sending a full tablet report...
I20260926 02:11:52.925763 4259 ts_tablet_manager.cc:1428] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da: Time spent starting tablet: real 0.017s user 0.006s sys 0.010s
I20260926 02:11:52.926680 4258 ts_tablet_manager.cc:1428] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260926 02:11:52.927237 4260 raft_consensus.cc:348] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eb7ca300bd4112926c8d81ed3292e1" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39241 } } peers { permanent_uuid: "d215f6ce0d924313809e6ca7bd14b0da" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 44443 } } peers { permanent_uuid: "9197eef629df49d3a2f9e9c698b2efe1" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 45147 } }
I20260926 02:11:52.927696 4260 raft_consensus.cc:374] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:11:52.927803 4260 raft_consensus.cc:729] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9197eef629df49d3a2f9e9c698b2efe1, State: Initialized, Role: FOLLOWER
I20260926 02:11:52.928480 4260 consensus_queue.cc:260] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eb7ca300bd4112926c8d81ed3292e1" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39241 } } peers { permanent_uuid: "d215f6ce0d924313809e6ca7bd14b0da" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 44443 } } peers { permanent_uuid: "9197eef629df49d3a2f9e9c698b2efe1" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 45147 } }
I20260926 02:11:52.931453 4260 ts_tablet_manager.cc:1428] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
W20260926 02:11:52.940140 3970 tablet.cc:2369] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:11:52.992738 4105 tablet.cc:2369] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:11:53.026206 4240 tablet.cc:2369] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:11:53.039628 4265 raft_consensus.cc:482] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:11:53.039901 4265 raft_consensus.cc:504] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eb7ca300bd4112926c8d81ed3292e1" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39241 } } peers { permanent_uuid: "d215f6ce0d924313809e6ca7bd14b0da" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 44443 } } peers { permanent_uuid: "9197eef629df49d3a2f9e9c698b2efe1" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 45147 } }
I20260926 02:11:53.041604 4265 leader_election.cc:288] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d215f6ce0d924313809e6ca7bd14b0da (127.18.159.131:44443), 9197eef629df49d3a2f9e9c698b2efe1 (127.18.159.129:45147)
I20260926 02:11:53.048753 4256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:53.041918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:56489 (local address 127.18.159.131:44443)
0926 02:11:53.042142 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:53.042152 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:53.042820 (+ 668us) server_negotiation.cc:407] Connection header received
0926 02:11:53.043031 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:53.043038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:53.043125 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:53.043216 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:11:53.043902 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:53.044791 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:53.046898 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:53.047677 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:53.047802 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:53.047917 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:53.048327 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 02:11:53.048472 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:11:53.048765 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:53.042321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:54595 (local address 127.18.159.129:45147)
0926 02:11:53.042532 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:53.042540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:53.043084 (+ 544us) server_negotiation.cc:407] Connection header received
0926 02:11:53.043266 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:53.043270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:53.043347 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:53.043452 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:11:53.044215 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:53.044939 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:53.046933 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:53.047769 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:53.047933 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:53.048018 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:53.048394 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 02:11:53.048521 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 02:11:53.048868 4268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:53.041807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:44443 (local address 127.18.159.130:56489)
0926 02:11:53.042700 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:53.042721 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:53.042888 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:53.043254 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:53.043259 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:53.043284 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:11:53.043727 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:53.043738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:53.044925 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:53.044930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:53.046703 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:53.046716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:53.047909 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:53.047916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:53.048114 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:53.048197 (+ 83us) client_negotiation.cc:769] Sending connection context
0926 02:11:53.048275 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 02:11:53.048426 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":97,"thread_start_us":750,"threads_started":1}
I20260926 02:11:53.049314 4267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:53.041910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:45147 (local address 127.18.159.130:54595)
0926 02:11:53.042966 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:53.042983 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:53.043106 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:53.043510 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:53.043514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:53.043536 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:11:53.043982 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:53.043994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:53.045099 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:53.045103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:53.046742 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:53.046754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:53.048007 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:53.048012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:53.048218 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:53.048241 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:11:53.048377 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 02:11:53.048499 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":143,"thread_start_us":624,"threads_started":1}
I20260926 02:11:53.049825 4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ce96cb3b4c426a9951533dabc0ce86" candidate_uuid: "23eb7ca300bd4112926c8d81ed3292e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d215f6ce0d924313809e6ca7bd14b0da" is_pre_election: true
I20260926 02:11:53.050312 4194 raft_consensus.cc:2454] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eb7ca300bd4112926c8d81ed3292e1 in term 0.
I20260926 02:11:53.050318 3924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ce96cb3b4c426a9951533dabc0ce86" candidate_uuid: "23eb7ca300bd4112926c8d81ed3292e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9197eef629df49d3a2f9e9c698b2efe1" is_pre_election: true
I20260926 02:11:53.050833 3924 raft_consensus.cc:2454] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eb7ca300bd4112926c8d81ed3292e1 in term 0.
I20260926 02:11:53.051025 3991 leader_election.cc:302] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eb7ca300bd4112926c8d81ed3292e1, d215f6ce0d924313809e6ca7bd14b0da; no voters:
I20260926 02:11:53.051462 4265 raft_consensus.cc:2781] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:11:53.051602 4265 raft_consensus.cc:482] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:11:53.051715 4265 raft_consensus.cc:3037] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:11:53.053942 4265 raft_consensus.cc:504] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eb7ca300bd4112926c8d81ed3292e1" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39241 } } peers { permanent_uuid: "d215f6ce0d924313809e6ca7bd14b0da" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 44443 } } peers { permanent_uuid: "9197eef629df49d3a2f9e9c698b2efe1" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 45147 } }
I20260926 02:11:53.054603 4265 leader_election.cc:288] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 [CANDIDATE]: Term 1 election: Requested vote from peers d215f6ce0d924313809e6ca7bd14b0da (127.18.159.131:44443), 9197eef629df49d3a2f9e9c698b2efe1 (127.18.159.129:45147)
I20260926 02:11:53.055048 4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ce96cb3b4c426a9951533dabc0ce86" candidate_uuid: "23eb7ca300bd4112926c8d81ed3292e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d215f6ce0d924313809e6ca7bd14b0da"
I20260926 02:11:53.055092 3924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ce96cb3b4c426a9951533dabc0ce86" candidate_uuid: "23eb7ca300bd4112926c8d81ed3292e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9197eef629df49d3a2f9e9c698b2efe1"
I20260926 02:11:53.055275 4194 raft_consensus.cc:3037] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:11:53.055302 3924 raft_consensus.cc:3037] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:11:53.058671 4194 raft_consensus.cc:2454] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eb7ca300bd4112926c8d81ed3292e1 in term 1.
I20260926 02:11:53.058674 3924 raft_consensus.cc:2454] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eb7ca300bd4112926c8d81ed3292e1 in term 1.
I20260926 02:11:53.059186 3992 leader_election.cc:302] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eb7ca300bd4112926c8d81ed3292e1, 9197eef629df49d3a2f9e9c698b2efe1; no voters:
I20260926 02:11:53.059623 4265 raft_consensus.cc:2781] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:11:53.060237 4265 raft_consensus.cc:686] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 [term 1 LEADER]: Becoming Leader. State: Replica: 23eb7ca300bd4112926c8d81ed3292e1, State: Running, Role: LEADER
I20260926 02:11:53.060837 4265 consensus_queue.cc:237] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eb7ca300bd4112926c8d81ed3292e1" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39241 } } peers { permanent_uuid: "d215f6ce0d924313809e6ca7bd14b0da" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 44443 } } peers { permanent_uuid: "9197eef629df49d3a2f9e9c698b2efe1" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 45147 } }
I20260926 02:11:53.066124 3779 catalog_manager.cc:5495] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23eb7ca300bd4112926c8d81ed3292e1 (127.18.159.130). New cstate: current_term: 1 leader_uuid: "23eb7ca300bd4112926c8d81ed3292e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eb7ca300bd4112926c8d81ed3292e1" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 39241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d215f6ce0d924313809e6ca7bd14b0da" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 44443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9197eef629df49d3a2f9e9c698b2efe1" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 45147 } health_report { overall_health: UNKNOWN } } }
I20260926 02:11:53.120756 4253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:53.115378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43672 (local address 127.18.159.130:39241)
0926 02:11:53.115572 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:53.115576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:53.115593 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:11:53.115677 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:53.115681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:53.115762 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:53.115887 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:11:53.116417 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:53.117157 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:53.118606 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:53.118991 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:53.119083 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:53.119558 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:11:53.120181 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:11:53.120286 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:53.120372 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 02:11:53.120438 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:11:53.129256 3924 raft_consensus.cc:1261] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1 [term 1 FOLLOWER]: Refusing update from remote peer 23eb7ca300bd4112926c8d81ed3292e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:11:53.129256 4194 raft_consensus.cc:1261] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da [term 1 FOLLOWER]: Refusing update from remote peer 23eb7ca300bd4112926c8d81ed3292e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:11:53.130420 4269 consensus_queue.cc:1035] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d215f6ce0d924313809e6ca7bd14b0da" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 44443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:11:53.130820 4265 consensus_queue.cc:1035] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9197eef629df49d3a2f9e9c698b2efe1" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:11:53.147384 4274 mvcc.cc:205] Tried to move back new op lower bound from 7333432168966344704 to 7333432168701714432. Current Snapshot: MvccSnapshot[applied={T|T < 7333432168966344704}]
I20260926 02:11:53.160092 3779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:11:53.160447 3779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:11:53.164559 3779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c63ce4710ee84751a2e26f05f5976508: Sending DeleteTablet for 3 replicas of tablet 02ce96cb3b4c426a9951533dabc0ce86
I20260926 02:11:53.166093 4174 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ce96cb3b4c426a9951533dabc0ce86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:11:53 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:39562
I20260926 02:11:53.166092 3904 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ce96cb3b4c426a9951533dabc0ce86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:11:53 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:42780
I20260926 02:11:53.167243 4039 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ce96cb3b4c426a9951533dabc0ce86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:11:53 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:43666
I20260926 02:11:53.168780 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 3840
I20260926 02:11:53.177330 4283 tablet_replica.cc:330] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da: stopping tablet replica
I20260926 02:11:53.177984 4283 raft_consensus.cc:2229] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:11:53.178678 4283 raft_consensus.cc:2258] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:11:53.180954 4282 tablet_replica.cc:330] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1: stopping tablet replica
I20260926 02:11:53.181581 4282 raft_consensus.cc:2229] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:11:53.182271 4282 raft_consensus.cc:2258] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:11:53.185118 4283 ts_tablet_manager.cc:1905] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:11:53.191154 4282 ts_tablet_manager.cc:1905] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:11:53.192410 4283 ts_tablet_manager.cc:1918] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:11:53.192605 4283 log.cc:1196] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/02ce96cb3b4c426a9951533dabc0ce86
I20260926 02:11:53.193159 4283 ts_tablet_manager.cc:1939] T 02ce96cb3b4c426a9951533dabc0ce86 P d215f6ce0d924313809e6ca7bd14b0da: Deleting consensus metadata
I20260926 02:11:53.195182 3763 catalog_manager.cc:4841] TS d215f6ce0d924313809e6ca7bd14b0da (127.18.159.131:44443): tablet 02ce96cb3b4c426a9951533dabc0ce86 (table test-table [id=ad66be8644e84581934f3d76dd6542ac]) successfully deleted
I20260926 02:11:53.197971 4284 tablet_replica.cc:330] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1: stopping tablet replica
I20260926 02:11:53.198503 4284 raft_consensus.cc:2229] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:11:53.198559 4282 ts_tablet_manager.cc:1918] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:11:53.198724 4282 log.cc:1196] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/02ce96cb3b4c426a9951533dabc0ce86
I20260926 02:11:53.199374 4282 ts_tablet_manager.cc:1939] T 02ce96cb3b4c426a9951533dabc0ce86 P 9197eef629df49d3a2f9e9c698b2efe1: Deleting consensus metadata
I20260926 02:11:53.199318 4284 raft_consensus.cc:2258] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:11:53.201363 3764 catalog_manager.cc:4841] TS 9197eef629df49d3a2f9e9c698b2efe1 (127.18.159.129:45147): tablet 02ce96cb3b4c426a9951533dabc0ce86 (table test-table [id=ad66be8644e84581934f3d76dd6542ac]) successfully deleted
I20260926 02:11:53.202322 4284 ts_tablet_manager.cc:1905] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 02:11:53 dist-test-slave-03rn krb5kdc[3733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:11:53.210144 4284 ts_tablet_manager.cc:1918] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:11:53.210381 4284 log.cc:1196] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/02ce96cb3b4c426a9951533dabc0ce86
I20260926 02:11:53.210974 4284 ts_tablet_manager.cc:1939] T 02ce96cb3b4c426a9951533dabc0ce86 P 23eb7ca300bd4112926c8d81ed3292e1: Deleting consensus metadata
I20260926 02:11:53.211619 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:53.170617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42788 (local address 127.18.159.129:45147)
0926 02:11:53.170851 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:53.170861 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:53.183234 (+ 12373us) server_negotiation.cc:407] Connection header received
0926 02:11:53.183767 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:53.183773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:53.183875 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:53.183953 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:53.185831 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:53.186954 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:53.196379 (+ 9425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:53.197123 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:53.197256 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:53.204826 (+ 7570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:53.204840 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:53.204854 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:53.204895 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:53.207901 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:53.208827 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:53.208835 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:53.208838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:53.208977 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:53.209388 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:53.209393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:53.209395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:53.209791 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:53.210127 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:53.210984 (+ 857us) server_negotiation.cc:299] Negotiation successful
0926 02:11:53.211332 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:11:53.212317 3964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:11:53.212951 3763 catalog_manager.cc:4841] TS 23eb7ca300bd4112926c8d81ed3292e1 (127.18.159.130:39241): tablet 02ce96cb3b4c426a9951533dabc0ce86 (table test-table [id=ad66be8644e84581934f3d76dd6542ac]) successfully deleted
I20260926 02:11:53.330375 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 3840
I20260926 02:11:53.349491 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 3975
Sep 26 02:11:53 dist-test-slave-03rn krb5kdc[3733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.130@KRBTEST.COM
I20260926 02:11:53.367852 4253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:53.350418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.18.159.130:39241)
0926 02:11:53.350584 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:53.350588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:53.350610 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:11:53.350704 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:53.350708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:53.350767 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:53.350851 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:53.352557 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:53.353384 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:53.355248 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:53.355782 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:53.355877 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:53.361121 (+ 5244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:53.361140 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:53.361155 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:53.361202 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:53.364447 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:53.365263 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:53.365276 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:53.365278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:53.365551 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:53.366080 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:53.366088 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:53.366093 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:53.366660 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:53.367032 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:53.367255 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 02:11:53.367483 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:11:53.368992 4099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:11:53.485752 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 3975
I20260926 02:11:53.504667 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 4110
Sep 26 02:11:53 dist-test-slave-03rn krb5kdc[3733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.131@KRBTEST.COM
I20260926 02:11:53.520246 4256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:53.505467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39572 (local address 127.18.159.131:44443)
0926 02:11:53.505678 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:53.505683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:53.505702 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:11:53.505812 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:53.505816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:53.505889 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:53.505975 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:53.507195 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:53.507945 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:53.509289 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:53.509675 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:53.509776 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:53.514401 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:53.514418 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:53.514434 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:53.514474 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:53.517232 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:53.518037 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:53.518048 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:53.518052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:53.518253 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:53.518692 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:53.518697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:53.518699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:53.519093 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:53.519350 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:53.519618 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 02:11:53.519869 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 02:11:53.520965 4234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:11:53.610992 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 4110
I20260926 02:11:53.627529 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 3747
I20260926 02:11:53.628701 3809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:11:53.741354 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 3747
2026-09-26T02:11:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4575 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T02:11:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:11:53Z Disabled control of system clock
I20260926 02:11:53.823567 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:34423
--webserver_interface=127.18.159.190
--webserver_port=0
--builtin_ntp_servers=127.18.159.148:43739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:34423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:11:54.168936 4293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:11:54.169237 4293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:11:54.169389 4293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:11:54.177047 4293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:11:54.177175 4293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:11:54.177258 4293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:11:54.177352 4293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:11:54.177441 4293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:11:54.187214 4293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:43739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:34423
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:34423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:11:54.188678 4293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:11:54.190546 4293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:11:54.203503 4301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:54.203609 4299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:54.203706 4298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:11:54.204408 4293 server_base.cc:1034] running on GCE node
I20260926 02:11:54.205263 4293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:11:54.206804 4293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:11:54.208033 4293 hybrid_clock.cc:648] HybridClock initialized: now 1790388714207990 us; error 33 us; skew 500 ppm
I20260926 02:11:54.208506 4293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:11:54.210835 4293 webserver.cc:460] Webserver started at http://127.18.159.190:37375/ using document root <none> and password file <none>
I20260926 02:11:54.211581 4293 fs_manager.cc:359] Metadata directory not provided
I20260926 02:11:54.211678 4293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:11:54.211970 4293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:11:54.215126 4293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "f9b5872297374c3c812209e14ad6cfe2"
format_stamp: "Formatted at 2026-09-26 02:11:54 on dist-test-slave-03rn"
server_key: "0e92d1df821382e17e65ac0ddf9ccccf"
server_key_iv: "b0046f692116cae341ef9d6acce7354e"
server_key_version: "encryptionkey@0"
I20260926 02:11:54.215952 4293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "f9b5872297374c3c812209e14ad6cfe2"
format_stamp: "Formatted at 2026-09-26 02:11:54 on dist-test-slave-03rn"
server_key: "0e92d1df821382e17e65ac0ddf9ccccf"
server_key_iv: "b0046f692116cae341ef9d6acce7354e"
server_key_version: "encryptionkey@0"
I20260926 02:11:54.221474 4293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:11:54.225520 4307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:54.227185 4293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:11:54.227373 4293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "f9b5872297374c3c812209e14ad6cfe2"
format_stamp: "Formatted at 2026-09-26 02:11:54 on dist-test-slave-03rn"
server_key: "0e92d1df821382e17e65ac0ddf9ccccf"
server_key_iv: "b0046f692116cae341ef9d6acce7354e"
server_key_version: "encryptionkey@0"
I20260926 02:11:54.227586 4293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:11:54.242857 4293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:11:54.243753 4293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:11:54.244014 4293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:11:54.264369 4293 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:34423
I20260926 02:11:54.264341 4360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:34423 every 8 connection(s)
I20260926 02:11:54.265841 4293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:11:54.269755 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 4293
I20260926 02:11:54.269961 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:11:54.270383 4361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:11:54.270511 19070 external_mini_cluster.cc:1428] Setting key 24b8fbf5a839a8cb544f8627f5b6e6e5
I20260926 02:11:54.284617 4361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2: Bootstrap starting.
I20260926 02:11:54.285457 4364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:54.272566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.18.159.190:34423)
0926 02:11:54.273698 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:54.273723 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:54.273816 (+ 93us) server_negotiation.cc:407] Connection header received
0926 02:11:54.275699 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:54.275730 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:54.276390 (+ 660us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:54.276869 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:54.277532 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:54.278925 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:54.280516 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:54.281278 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:54.281544 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:54.282817 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:54.282825 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:54.282854 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:11:54.283476 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:54.283560 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:11:54.283583 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:54.283725 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:54.283975 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 02:11:54.284361 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":377,"threads_started":1}
I20260926 02:11:54.288756 4361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2: Neither blocks nor log segments found. Creating new log.
I20260926 02:11:54.289856 4361 log.cc:824] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2: Log is configured to *not* fsync() on all Append() calls
I20260926 02:11:54.293170 4361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2: No bootstrap required, opened a new log
I20260926 02:11:54.301167 4361 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b5872297374c3c812209e14ad6cfe2" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 34423 } }
I20260926 02:11:54.301579 4361 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:11:54.301657 4361 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b5872297374c3c812209e14ad6cfe2, State: Initialized, Role: FOLLOWER
I20260926 02:11:54.302253 4361 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b5872297374c3c812209e14ad6cfe2" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 34423 } }
I20260926 02:11:54.302481 4361 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:11:54.302583 4361 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:11:54.302732 4361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:11:54.305328 4361 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b5872297374c3c812209e14ad6cfe2" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 34423 } }
I20260926 02:11:54.305938 4361 leader_election.cc:302] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9b5872297374c3c812209e14ad6cfe2; no voters:
I20260926 02:11:54.306581 4361 leader_election.cc:288] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:11:54.306779 4366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:11:54.307751 4366 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2 [term 1 LEADER]: Becoming Leader. State: Replica: f9b5872297374c3c812209e14ad6cfe2, State: Running, Role: LEADER
I20260926 02:11:54.308394 4366 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b5872297374c3c812209e14ad6cfe2" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 34423 } }
I20260926 02:11:54.308887 4361 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:11:54.311710 4367 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9b5872297374c3c812209e14ad6cfe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b5872297374c3c812209e14ad6cfe2" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 34423 } } }
I20260926 02:11:54.311818 4368 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9b5872297374c3c812209e14ad6cfe2. Latest consensus state: current_term: 1 leader_uuid: "f9b5872297374c3c812209e14ad6cfe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b5872297374c3c812209e14ad6cfe2" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 34423 } } }
I20260926 02:11:54.312358 4368 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2 [sys.catalog]: This master's current role is: LEADER
I20260926 02:11:54.312364 4367 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2 [sys.catalog]: This master's current role is: LEADER
I20260926 02:11:54.317183 4374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:11:54.322274 4374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:11:54.332381 4374 catalog_manager.cc:1347] Generated new cluster ID: 4a576bfa65a84a4cabfd98fb329f51af
I20260926 02:11:54.332516 4374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:11:54.348315 4374 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:11:54.349628 4374 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:11:54.357152 4374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2: Generated new TSK 0
I20260926 02:11:54.358259 4374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:11:54.376971 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:34423
--builtin_ntp_servers=127.18.159.148:43739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:11:54.734704 4385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:11:54.734990 4385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:11:54.735137 4385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:11:54.742690 4385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:11:54.742815 4385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:11:54.742988 4385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:11:54.752709 4385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:43739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:34423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:11:54.754163 4385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:11:54.755972 4385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:11:54.769984 4391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:54.770084 4390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:54.770027 4393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:11:54.771175 4385 server_base.cc:1034] running on GCE node
I20260926 02:11:54.771890 4385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:11:54.773041 4385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:11:54.774289 4385 hybrid_clock.cc:648] HybridClock initialized: now 1790388714774220 us; error 62 us; skew 500 ppm
I20260926 02:11:54.774725 4385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:11:54.777217 4385 webserver.cc:460] Webserver started at http://127.18.159.129:45517/ using document root <none> and password file <none>
I20260926 02:11:54.777956 4385 fs_manager.cc:359] Metadata directory not provided
I20260926 02:11:54.778090 4385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:11:54.778409 4385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:11:54.781420 4385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "74232a70573049c6b7e29552e3394b42"
format_stamp: "Formatted at 2026-09-26 02:11:54 on dist-test-slave-03rn"
server_key: "cf1c04a380c09473c13b8b00a139d9a5"
server_key_iv: "170dafe3cfd693eae03c489c4b4841e2"
server_key_version: "encryptionkey@0"
I20260926 02:11:54.782287 4385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "74232a70573049c6b7e29552e3394b42"
format_stamp: "Formatted at 2026-09-26 02:11:54 on dist-test-slave-03rn"
server_key: "cf1c04a380c09473c13b8b00a139d9a5"
server_key_iv: "170dafe3cfd693eae03c489c4b4841e2"
server_key_version: "encryptionkey@0"
I20260926 02:11:54.787719 4385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:11:54.791796 4399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:54.793428 4385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:11:54.793669 4385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "74232a70573049c6b7e29552e3394b42"
format_stamp: "Formatted at 2026-09-26 02:11:54 on dist-test-slave-03rn"
server_key: "cf1c04a380c09473c13b8b00a139d9a5"
server_key_iv: "170dafe3cfd693eae03c489c4b4841e2"
server_key_version: "encryptionkey@0"
I20260926 02:11:54.793924 4385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:11:54.809800 4385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:11:54.810746 4385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:11:54.811076 4385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:11:54.812098 4385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:11:54.813627 4385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:11:54.813747 4385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:54.813880 4385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:11:54.813967 4385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:54.849030 4385 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:36431
I20260926 02:11:54.849085 4512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:36431 every 8 connection(s)
I20260926 02:11:54.850440 4385 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:11:54.855913 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 4385
I20260926 02:11:54.856115 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:11:54.856588 19070 external_mini_cluster.cc:1428] Setting key e5362e89aaeabe59eb11a12a8b13f38f
I20260926 02:11:54.859499 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:34423
--builtin_ntp_servers=127.18.159.148:43739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:11:54.865526 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:54.853194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:59499 (local address 127.18.159.190:34423)
0926 02:11:54.853702 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:54.853706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:54.854773 (+ 1067us) server_negotiation.cc:407] Connection header received
0926 02:11:54.857089 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:54.857095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:54.857188 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:54.857311 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:54.858741 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:54.859448 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:54.862387 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:54.862817 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:54.862922 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:54.864707 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:54.864712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:54.864715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:11:54.864736 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:54.864775 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:11:54.864786 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:54.864858 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:54.865147 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 02:11:54.865238 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":295,"threads_started":1}
I20260926 02:11:54.867230 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:54.853695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:34423 (local address 127.18.159.129:59499)
0926 02:11:54.854610 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:54.854646 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:54.856775 (+ 2129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:54.857597 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:54.857612 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:54.857743 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:11:54.858556 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:54.858584 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:54.859641 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:54.859648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:54.862219 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:54.862239 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:54.863076 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:54.863084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:54.863448 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:54.864310 (+ 862us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:11:54.864315 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:11:54.864376 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:11:54.864491 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:11:54.864896 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:11:54.864912 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:11:54.864924 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:11:54.865119 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 02:11:54.865608 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":286,"threads_started":1}
I20260926 02:11:54.869012 4513 heartbeater.cc:347] Connected to a master server at 127.18.159.190:34423
I20260926 02:11:54.869351 4513 heartbeater.cc:464] Registering TS with master...
I20260926 02:11:54.870319 4513 heartbeater.cc:511] Master 127.18.159.190:34423 requested a full tablet report, sending...
I20260926 02:11:54.872759 4325 ts_manager.cc:194] Registered new tserver with Master: 74232a70573049c6b7e29552e3394b42 (127.18.159.129:36431)
I20260926 02:11:54.874217 4325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.159.129:59499
W20260926 02:11:55.207121 4517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:11:55.207432 4517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:11:55.207589 4517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:11:55.215405 4517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:11:55.215531 4517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:11:55.215731 4517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:11:55.225227 4517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:43739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:34423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:11:55.226600 4517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:11:55.228477 4517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:11:55.242241 4522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:55.242308 4523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:55.242336 4525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:11:55.243110 4517 server_base.cc:1034] running on GCE node
I20260926 02:11:55.243904 4517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:11:55.245091 4517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:11:55.246417 4517 hybrid_clock.cc:648] HybridClock initialized: now 1790388715246306 us; error 56 us; skew 500 ppm
I20260926 02:11:55.246794 4517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:11:55.249485 4517 webserver.cc:460] Webserver started at http://127.18.159.130:35453/ using document root <none> and password file <none>
I20260926 02:11:55.250277 4517 fs_manager.cc:359] Metadata directory not provided
I20260926 02:11:55.250429 4517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:11:55.250766 4517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:11:55.253940 4517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "f1a91475f17740ee85eaa5f25da6a617"
format_stamp: "Formatted at 2026-09-26 02:11:55 on dist-test-slave-03rn"
server_key: "6c9b227171c524864b73179fa7dcc01c"
server_key_iv: "1fd617694cf72691a7298175a5fe9212"
server_key_version: "encryptionkey@0"
I20260926 02:11:55.254817 4517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "f1a91475f17740ee85eaa5f25da6a617"
format_stamp: "Formatted at 2026-09-26 02:11:55 on dist-test-slave-03rn"
server_key: "6c9b227171c524864b73179fa7dcc01c"
server_key_iv: "1fd617694cf72691a7298175a5fe9212"
server_key_version: "encryptionkey@0"
I20260926 02:11:55.260622 4517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 02:11:55.265141 4531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:55.266853 4517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260926 02:11:55.267108 4517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "f1a91475f17740ee85eaa5f25da6a617"
format_stamp: "Formatted at 2026-09-26 02:11:55 on dist-test-slave-03rn"
server_key: "6c9b227171c524864b73179fa7dcc01c"
server_key_iv: "1fd617694cf72691a7298175a5fe9212"
server_key_version: "encryptionkey@0"
I20260926 02:11:55.267369 4517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:11:55.294634 4517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:11:55.295571 4517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:11:55.295904 4517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:11:55.297048 4517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:11:55.298708 4517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:11:55.298834 4517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:55.298965 4517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:11:55.299065 4517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:55.334422 4517 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:45397
I20260926 02:11:55.334518 4644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:45397 every 8 connection(s)
I20260926 02:11:55.335758 4517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:11:55.339356 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 4517
I20260926 02:11:55.339548 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:11:55.340000 19070 external_mini_cluster.cc:1428] Setting key 46b1085b5bef0eac61593db58df6ea36
I20260926 02:11:55.343093 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:34423
--builtin_ntp_servers=127.18.159.148:43739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:11:55.350004 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:55.338387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:35245 (local address 127.18.159.190:34423)
0926 02:11:55.338609 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:55.338615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:55.340383 (+ 1768us) server_negotiation.cc:407] Connection header received
0926 02:11:55.342499 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:55.342504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:55.342590 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:55.342702 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:55.343892 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.345098 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.346703 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.347429 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.347594 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:55.349055 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:55.349064 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:55.349071 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:11:55.349102 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:55.349148 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:11:55.349167 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:55.349265 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:55.349482 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 02:11:55.349600 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:11:55.350970 4647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:55.339012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:34423 (local address 127.18.159.130:35245)
0926 02:11:55.340189 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:55.340236 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:55.342190 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:55.342908 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:55.342917 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:55.343043 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:11:55.343715 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:55.343733 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.345239 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.345243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:55.346542 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:55.346556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.347662 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.347667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:55.347878 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:55.348699 (+ 821us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:11:55.348702 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:11:55.348751 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:11:55.348843 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:11:55.349266 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:11:55.349274 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:11:55.349280 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 02:11:55.349433 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 02:11:55.349762 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":412,"threads_started":1}
I20260926 02:11:55.352746 4645 heartbeater.cc:347] Connected to a master server at 127.18.159.190:34423
I20260926 02:11:55.353121 4645 heartbeater.cc:464] Registering TS with master...
I20260926 02:11:55.353957 4645 heartbeater.cc:511] Master 127.18.159.190:34423 requested a full tablet report, sending...
I20260926 02:11:55.355575 4325 ts_manager.cc:194] Registered new tserver with Master: f1a91475f17740ee85eaa5f25da6a617 (127.18.159.130:45397)
I20260926 02:11:55.356889 4325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.159.130:35245
W20260926 02:11:55.686846 4648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:11:55.687144 4648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:11:55.687229 4648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:11:55.694543 4648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:11:55.694669 4648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:11:55.694777 4648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:11:55.704216 4648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:43739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:34423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:11:55.705497 4648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:11:55.707307 4648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:11:55.721283 4653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:55.721253 4656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:55.721347 4654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:11:55.727290 4648 server_base.cc:1034] running on GCE node
I20260926 02:11:55.728293 4648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:11:55.729562 4648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:11:55.730912 4648 hybrid_clock.cc:648] HybridClock initialized: now 1790388715730835 us; error 62 us; skew 500 ppm
I20260926 02:11:55.731328 4648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:11:55.734221 4648 webserver.cc:460] Webserver started at http://127.18.159.131:45933/ using document root <none> and password file <none>
I20260926 02:11:55.735038 4648 fs_manager.cc:359] Metadata directory not provided
I20260926 02:11:55.735167 4648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:11:55.735530 4648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:11:55.738988 4648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "5bf139896e7d4f3b8e7648413e5a0fdb"
format_stamp: "Formatted at 2026-09-26 02:11:55 on dist-test-slave-03rn"
server_key: "42643a6cc63d018c57bae09ce1472082"
server_key_iv: "8f3bf717a8fb6a69735a7e5a88bd98e1"
server_key_version: "encryptionkey@0"
I20260926 02:11:55.739975 4648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "5bf139896e7d4f3b8e7648413e5a0fdb"
format_stamp: "Formatted at 2026-09-26 02:11:55 on dist-test-slave-03rn"
server_key: "42643a6cc63d018c57bae09ce1472082"
server_key_iv: "8f3bf717a8fb6a69735a7e5a88bd98e1"
server_key_version: "encryptionkey@0"
I20260926 02:11:55.746310 4648 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 02:11:55.750586 4662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:55.752465 4648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:11:55.752715 4648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "5bf139896e7d4f3b8e7648413e5a0fdb"
format_stamp: "Formatted at 2026-09-26 02:11:55 on dist-test-slave-03rn"
server_key: "42643a6cc63d018c57bae09ce1472082"
server_key_iv: "8f3bf717a8fb6a69735a7e5a88bd98e1"
server_key_version: "encryptionkey@0"
I20260926 02:11:55.753157 4648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:11:55.769359 4648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:11:55.770337 4648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:11:55.770736 4648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:11:55.772219 4648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:11:55.774379 4648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:11:55.774699 4648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:55.774964 4648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:11:55.775115 4648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:55.812968 4648 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:37997
I20260926 02:11:55.813035 4776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:37997 every 8 connection(s)
I20260926 02:11:55.814421 4648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:11:55.821484 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 4648
I20260926 02:11:55.821674 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:11:55.822134 19070 external_mini_cluster.cc:1428] Setting key 684e1046ec172ba67d90cab6cb6d0aa8
I20260926 02:11:55.828037 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:55.817387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:46979 (local address 127.18.159.190:34423)
0926 02:11:55.817573 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:55.817577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:55.819013 (+ 1436us) server_negotiation.cc:407] Connection header received
0926 02:11:55.820893 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:55.820898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:55.821000 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:55.821112 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:55.822295 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.823265 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.825495 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.825882 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.825987 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:55.827290 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:55.827295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:55.827298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:11:55.827315 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:55.827346 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:11:55.827358 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:55.827430 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:55.827703 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 02:11:55.827785 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:11:55.829154 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:55.817876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:34423 (local address 127.18.159.131:46979)
0926 02:11:55.818817 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:55.818863 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:55.820622 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:55.821354 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:55.821363 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:55.821453 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:11:55.822119 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:55.822136 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.823542 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.823548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:55.825358 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:55.825371 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.826064 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.826068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:55.826268 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:55.826983 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:11:55.826985 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:11:55.827027 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:11:55.827110 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:11:55.827471 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:11:55.827481 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:11:55.827488 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 02:11:55.827673 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 02:11:55.828003 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":286,"threads_started":1}
I20260926 02:11:55.830703 4777 heartbeater.cc:347] Connected to a master server at 127.18.159.190:34423
I20260926 02:11:55.831087 4777 heartbeater.cc:464] Registering TS with master...
I20260926 02:11:55.831914 4777 heartbeater.cc:511] Master 127.18.159.190:34423 requested a full tablet report, sending...
I20260926 02:11:55.833474 4325 ts_manager.cc:194] Registered new tserver with Master: 5bf139896e7d4f3b8e7648413e5a0fdb (127.18.159.131:37997)
I20260926 02:11:55.834316 4325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.159.131:46979
I20260926 02:11:55.837656 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:11:55.847535 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:55.841719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44022 (local address 127.18.159.190:34423)
0926 02:11:55.841886 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:55.841890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:55.842390 (+ 500us) server_negotiation.cc:407] Connection header received
0926 02:11:55.842536 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:55.842540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:55.842604 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:55.842692 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:55.843161 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.843860 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.845297 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.845701 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.845797 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:55.846921 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:55.846926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:55.846929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:11:55.846947 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:55.846994 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:11:55.847006 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:55.847074 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:55.847215 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 02:11:55.847292 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:11:55.851714 4325 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:11:55.858799 4325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:11:55.878309 4513 heartbeater.cc:503] Master 127.18.159.190:34423 was elected leader, sending a full tablet report...
I20260926 02:11:55.894483 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:55.871312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:45397 (local address 127.0.0.1:56682)
0926 02:11:55.872026 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:55.872072 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:55.872258 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:55.884666 (+ 12408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:55.884678 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:55.884730 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:11:55.885317 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:55.885336 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.889184 (+ 3848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.889192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:55.891846 (+ 2654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:55.891860 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.893465 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.893471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:55.893684 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:55.893736 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 02:11:55.893870 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 02:11:55.894103 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":503,"threads_started":1}
I20260926 02:11:55.894546 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:55.871698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.18.159.130:45397)
0926 02:11:55.883400 (+ 11702us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:55.883413 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:55.883457 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:11:55.883622 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:55.883634 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:55.884018 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:55.885731 (+ 1713us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:11:55.886017 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.888029 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.892172 (+ 4143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.893248 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.893449 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:55.893708 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:55.894053 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 02:11:55.894232 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11483,"thread_start_us":12573,"threads_started":1}
I20260926 02:11:55.899567 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:55.874184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:37997 (local address 127.0.0.1:54140)
0926 02:11:55.890770 (+ 16586us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:55.890794 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:55.890937 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:55.891801 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:55.891806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:55.891833 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:11:55.892284 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:55.892296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.894308 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.894315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:55.896929 (+ 2614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:55.896944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.898522 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.898528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:55.898857 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:55.898882 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 02:11:55.898989 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 02:11:55.899162 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16462,"mutex_wait_us":3,"thread_start_us":16842,"threads_started":1}
I20260926 02:11:55.898962 4580 tablet_service.cc:1467] Processing CreateTablet for tablet cdcb885c490f45258325013bee177630 (DEFAULT_TABLE table=test-table [id=3e3e137b869f451ea6d092a9cf691fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:11:55.899643 4794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:55.874818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54140 (local address 127.18.159.131:37997)
0926 02:11:55.883025 (+ 8207us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:55.883040 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:55.890960 (+ 7920us) server_negotiation.cc:407] Connection header received
0926 02:11:55.891128 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:55.891143 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:55.891525 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:55.891793 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:11:55.892474 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.894125 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.897158 (+ 3033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.898306 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.898518 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:55.898736 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:55.899049 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 02:11:55.899217 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7949,"thread_start_us":7749,"threads_started":1}
I20260926 02:11:55.900662 4580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdcb885c490f45258325013bee177630. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:11:55.902205 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:55.871446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:36431 (local address 127.0.0.1:33540)
0926 02:11:55.888804 (+ 17358us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:55.888838 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:55.888989 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:55.894015 (+ 5026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:55.894020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:55.894047 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:11:55.894527 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:55.894541 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.897114 (+ 2573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.897119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:55.899096 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:55.899112 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.901361 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.901367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:55.901554 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:55.901587 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 02:11:55.901687 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 02:11:55.901828 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17143,"mutex_wait_us":1,"thread_start_us":16934,"threads_started":1}
I20260926 02:11:55.902297 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:55.871801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33540 (local address 127.18.159.129:36431)
0926 02:11:55.893383 (+ 21582us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:55.893392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:55.893424 (+ 32us) server_negotiation.cc:407] Connection header received
0926 02:11:55.893523 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:55.893531 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:55.893782 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:55.894655 (+ 873us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:11:55.895086 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.896954 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.899351 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:55.900833 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:55.901042 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:55.901235 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:55.901771 (+ 536us) server_negotiation.cc:299] Negotiation successful
0926 02:11:55.901954 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21389,"thread_start_us":21917,"threads_started":1}
I20260926 02:11:55.903738 4712 tablet_service.cc:1467] Processing CreateTablet for tablet cdcb885c490f45258325013bee177630 (DEFAULT_TABLE table=test-table [id=3e3e137b869f451ea6d092a9cf691fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:11:55.905212 4712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdcb885c490f45258325013bee177630. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:11:55.908140 4447 tablet_service.cc:1467] Processing CreateTablet for tablet cdcb885c490f45258325013bee177630 (DEFAULT_TABLE table=test-table [id=3e3e137b869f451ea6d092a9cf691fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:11:55.909698 4447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdcb885c490f45258325013bee177630. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:11:55.916054 4795 tablet_bootstrap.cc:492] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617: Bootstrap starting.
I20260926 02:11:55.918133 4796 tablet_bootstrap.cc:492] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb: Bootstrap starting.
I20260926 02:11:55.920060 4795 tablet_bootstrap.cc:654] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617: Neither blocks nor log segments found. Creating new log.
I20260926 02:11:55.921120 4795 log.cc:824] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617: Log is configured to *not* fsync() on all Append() calls
I20260926 02:11:55.921741 4796 tablet_bootstrap.cc:654] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb: Neither blocks nor log segments found. Creating new log.
I20260926 02:11:55.922941 4796 log.cc:824] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb: Log is configured to *not* fsync() on all Append() calls
I20260926 02:11:55.924851 4795 tablet_bootstrap.cc:492] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617: No bootstrap required, opened a new log
I20260926 02:11:55.925120 4795 ts_tablet_manager.cc:1397] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 02:11:55.928541 4799 tablet_bootstrap.cc:492] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42: Bootstrap starting.
I20260926 02:11:55.931865 4799 tablet_bootstrap.cc:654] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42: Neither blocks nor log segments found. Creating new log.
I20260926 02:11:55.933205 4799 log.cc:824] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42: Log is configured to *not* fsync() on all Append() calls
I20260926 02:11:55.933153 4795 raft_consensus.cc:348] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a91475f17740ee85eaa5f25da6a617" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45397 } } peers { permanent_uuid: "74232a70573049c6b7e29552e3394b42" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 36431 } } peers { permanent_uuid: "5bf139896e7d4f3b8e7648413e5a0fdb" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37997 } }
I20260926 02:11:55.933591 4795 raft_consensus.cc:374] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:11:55.933713 4795 raft_consensus.cc:729] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a91475f17740ee85eaa5f25da6a617, State: Initialized, Role: FOLLOWER
I20260926 02:11:55.933982 4796 tablet_bootstrap.cc:492] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb: No bootstrap required, opened a new log
I20260926 02:11:55.934283 4796 ts_tablet_manager.cc:1397] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20260926 02:11:55.934425 4795 consensus_queue.cc:260] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a91475f17740ee85eaa5f25da6a617" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45397 } } peers { permanent_uuid: "74232a70573049c6b7e29552e3394b42" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 36431 } } peers { permanent_uuid: "5bf139896e7d4f3b8e7648413e5a0fdb" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37997 } }
I20260926 02:11:55.935593 4645 heartbeater.cc:503] Master 127.18.159.190:34423 was elected leader, sending a full tablet report...
I20260926 02:11:55.936277 4795 ts_tablet_manager.cc:1428] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260926 02:11:55.937837 4799 tablet_bootstrap.cc:492] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42: No bootstrap required, opened a new log
I20260926 02:11:55.938128 4799 ts_tablet_manager.cc:1397] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260926 02:11:55.943822 4796 raft_consensus.cc:348] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a91475f17740ee85eaa5f25da6a617" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45397 } } peers { permanent_uuid: "74232a70573049c6b7e29552e3394b42" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 36431 } } peers { permanent_uuid: "5bf139896e7d4f3b8e7648413e5a0fdb" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37997 } }
I20260926 02:11:55.944380 4796 raft_consensus.cc:374] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:11:55.944509 4796 raft_consensus.cc:729] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf139896e7d4f3b8e7648413e5a0fdb, State: Initialized, Role: FOLLOWER
I20260926 02:11:55.945173 4796 consensus_queue.cc:260] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a91475f17740ee85eaa5f25da6a617" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45397 } } peers { permanent_uuid: "74232a70573049c6b7e29552e3394b42" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 36431 } } peers { permanent_uuid: "5bf139896e7d4f3b8e7648413e5a0fdb" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37997 } }
I20260926 02:11:55.946296 4777 heartbeater.cc:503] Master 127.18.159.190:34423 was elected leader, sending a full tablet report...
I20260926 02:11:55.946450 4799 raft_consensus.cc:348] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a91475f17740ee85eaa5f25da6a617" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45397 } } peers { permanent_uuid: "74232a70573049c6b7e29552e3394b42" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 36431 } } peers { permanent_uuid: "5bf139896e7d4f3b8e7648413e5a0fdb" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37997 } }
I20260926 02:11:55.946902 4796 ts_tablet_manager.cc:1428] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb: Time spent starting tablet: real 0.012s user 0.011s sys 0.003s
I20260926 02:11:55.946919 4799 raft_consensus.cc:374] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:11:55.947016 4799 raft_consensus.cc:729] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74232a70573049c6b7e29552e3394b42, State: Initialized, Role: FOLLOWER
I20260926 02:11:55.947671 4799 consensus_queue.cc:260] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a91475f17740ee85eaa5f25da6a617" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45397 } } peers { permanent_uuid: "74232a70573049c6b7e29552e3394b42" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 36431 } } peers { permanent_uuid: "5bf139896e7d4f3b8e7648413e5a0fdb" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37997 } }
I20260926 02:11:55.949527 4799 ts_tablet_manager.cc:1428] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260926 02:11:56.052285 4803 raft_consensus.cc:482] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:11:56.052649 4803 raft_consensus.cc:504] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a91475f17740ee85eaa5f25da6a617" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45397 } } peers { permanent_uuid: "74232a70573049c6b7e29552e3394b42" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 36431 } } peers { permanent_uuid: "5bf139896e7d4f3b8e7648413e5a0fdb" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37997 } }
I20260926 02:11:56.054576 4803 leader_election.cc:288] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a91475f17740ee85eaa5f25da6a617 (127.18.159.130:45397), 5bf139896e7d4f3b8e7648413e5a0fdb (127.18.159.131:37997)
I20260926 02:11:56.061887 4794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:56.055431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:42599 (local address 127.18.159.131:37997)
0926 02:11:56.055632 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:56.055637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:56.056217 (+ 580us) server_negotiation.cc:407] Connection header received
0926 02:11:56.056436 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:56.056441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:56.056543 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:56.056647 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:11:56.057289 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:56.058102 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:56.060084 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:56.060931 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:56.061119 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:56.061190 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:56.061534 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 02:11:56.061666 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:11:56.061883 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:56.054958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:50295 (local address 127.18.159.130:45397)
0926 02:11:56.055169 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:56.055178 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:56.056000 (+ 822us) server_negotiation.cc:407] Connection header received
0926 02:11:56.056236 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:56.056241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:56.056344 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:56.056438 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:11:56.057108 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:56.057999 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:56.060135 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:56.060891 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:56.061039 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:56.061118 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:56.061471 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 02:11:56.061612 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:11:56.062004 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:56.054961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:45397 (local address 127.18.159.129:50295)
0926 02:11:56.055851 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:56.055876 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:56.056055 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:56.056466 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:56.056471 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:56.056498 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:11:56.056919 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:56.056933 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:56.058127 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:56.058135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:56.059895 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:56.059907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:56.061085 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:56.061091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:56.061266 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:56.061341 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 02:11:56.061414 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 02:11:56.061558 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":88,"thread_start_us":418,"threads_started":1}
I20260926 02:11:56.062055 4805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:56.055107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:37997 (local address 127.18.159.129:42599)
0926 02:11:56.056086 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:56.056100 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:56.056283 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:56.056684 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:56.056687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:56.056707 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:11:56.057132 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:56.057144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:56.058255 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:56.058259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:56.059893 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:56.059905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:56.061171 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:56.061177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:56.061354 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:56.061376 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 02:11:56.061490 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 02:11:56.061621 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":109,"thread_start_us":642,"threads_started":1}
I20260926 02:11:56.063048 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdcb885c490f45258325013bee177630" candidate_uuid: "74232a70573049c6b7e29552e3394b42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a91475f17740ee85eaa5f25da6a617" is_pre_election: true
I20260926 02:11:56.063397 4732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdcb885c490f45258325013bee177630" candidate_uuid: "74232a70573049c6b7e29552e3394b42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf139896e7d4f3b8e7648413e5a0fdb" is_pre_election: true
I20260926 02:11:56.063695 4600 raft_consensus.cc:2454] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74232a70573049c6b7e29552e3394b42 in term 0.
I20260926 02:11:56.063959 4732 raft_consensus.cc:2454] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74232a70573049c6b7e29552e3394b42 in term 0.
I20260926 02:11:56.064487 4400 leader_election.cc:302] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74232a70573049c6b7e29552e3394b42, f1a91475f17740ee85eaa5f25da6a617; no voters:
I20260926 02:11:56.064982 4803 raft_consensus.cc:2781] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:11:56.065138 4803 raft_consensus.cc:482] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:11:56.065250 4803 raft_consensus.cc:3037] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 [term 0 FOLLOWER]: Advancing to term 1
W20260926 02:11:56.066316 4778 tablet.cc:2369] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:11:56.067505 4803 raft_consensus.cc:504] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a91475f17740ee85eaa5f25da6a617" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45397 } } peers { permanent_uuid: "74232a70573049c6b7e29552e3394b42" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 36431 } } peers { permanent_uuid: "5bf139896e7d4f3b8e7648413e5a0fdb" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37997 } }
I20260926 02:11:56.068249 4803 leader_election.cc:288] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 [CANDIDATE]: Term 1 election: Requested vote from peers f1a91475f17740ee85eaa5f25da6a617 (127.18.159.130:45397), 5bf139896e7d4f3b8e7648413e5a0fdb (127.18.159.131:37997)
I20260926 02:11:56.068831 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdcb885c490f45258325013bee177630" candidate_uuid: "74232a70573049c6b7e29552e3394b42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a91475f17740ee85eaa5f25da6a617"
I20260926 02:11:56.068853 4732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdcb885c490f45258325013bee177630" candidate_uuid: "74232a70573049c6b7e29552e3394b42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf139896e7d4f3b8e7648413e5a0fdb"
I20260926 02:11:56.069048 4600 raft_consensus.cc:3037] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:11:56.069114 4732 raft_consensus.cc:3037] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:11:56.071631 4600 raft_consensus.cc:2454] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74232a70573049c6b7e29552e3394b42 in term 1.
I20260926 02:11:56.071767 4732 raft_consensus.cc:2454] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74232a70573049c6b7e29552e3394b42 in term 1.
I20260926 02:11:56.072204 4400 leader_election.cc:302] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74232a70573049c6b7e29552e3394b42, f1a91475f17740ee85eaa5f25da6a617; no voters:
I20260926 02:11:56.072525 4803 raft_consensus.cc:2781] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:11:56.073046 4803 raft_consensus.cc:686] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 [term 1 LEADER]: Becoming Leader. State: Replica: 74232a70573049c6b7e29552e3394b42, State: Running, Role: LEADER
I20260926 02:11:56.073585 4803 consensus_queue.cc:237] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a91475f17740ee85eaa5f25da6a617" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45397 } } peers { permanent_uuid: "74232a70573049c6b7e29552e3394b42" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 36431 } } peers { permanent_uuid: "5bf139896e7d4f3b8e7648413e5a0fdb" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37997 } }
I20260926 02:11:56.078671 4324 catalog_manager.cc:5495] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74232a70573049c6b7e29552e3394b42 (127.18.159.129). New cstate: current_term: 1 leader_uuid: "74232a70573049c6b7e29552e3394b42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a91475f17740ee85eaa5f25da6a617" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74232a70573049c6b7e29552e3394b42" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 36431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bf139896e7d4f3b8e7648413e5a0fdb" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37997 } health_report { overall_health: UNKNOWN } } }
W20260926 02:11:56.087972 4646 tablet.cc:2369] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:11:56.103020 4514 tablet.cc:2369] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:11:56.124248 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:56.118312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33554 (local address 127.18.159.129:36431)
0926 02:11:56.118567 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:56.118574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:56.118610 (+ 36us) server_negotiation.cc:407] Connection header received
0926 02:11:56.118727 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:56.118731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:56.118839 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:56.119009 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:11:56.119431 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:56.120275 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:56.121811 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:56.122284 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:56.122437 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:56.122991 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:11:56.123567 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:11:56.123740 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:56.123803 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 02:11:56.123898 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:11:56.133386 4732 raft_consensus.cc:1261] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb [term 1 FOLLOWER]: Refusing update from remote peer 74232a70573049c6b7e29552e3394b42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:11:56.133543 4600 raft_consensus.cc:1261] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617 [term 1 FOLLOWER]: Refusing update from remote peer 74232a70573049c6b7e29552e3394b42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:11:56.134531 4806 consensus_queue.cc:1035] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf139896e7d4f3b8e7648413e5a0fdb" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 37997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:11:56.134977 4803 consensus_queue.cc:1035] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a91475f17740ee85eaa5f25da6a617" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:11:56.153674 4810 mvcc.cc:205] Tried to move back new op lower bound from 7333432181270831104 to 7333432181041324032. Current Snapshot: MvccSnapshot[applied={T|T < 7333432181270831104}]
I20260926 02:11:56.163770 4324 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:11:56.164106 4324 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:11:56.168370 4324 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9b5872297374c3c812209e14ad6cfe2: Sending DeleteTablet for 3 replicas of tablet cdcb885c490f45258325013bee177630
I20260926 02:11:56.169670 4712 tablet_service.cc:1514] Processing DeleteTablet for tablet cdcb885c490f45258325013bee177630 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:11:56 UTC) from {username='slave'} at 127.0.0.1:54140
I20260926 02:11:56.169687 4580 tablet_service.cc:1514] Processing DeleteTablet for tablet cdcb885c490f45258325013bee177630 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:11:56 UTC) from {username='slave'} at 127.0.0.1:56682
I20260926 02:11:56.170306 4447 tablet_service.cc:1514] Processing DeleteTablet for tablet cdcb885c490f45258325013bee177630 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:11:56 UTC) from {username='slave'} at 127.0.0.1:33540
I20260926 02:11:56.174324 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 4385
I20260926 02:11:56.182574 4819 tablet_replica.cc:330] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617: stopping tablet replica
I20260926 02:11:56.182845 4820 tablet_replica.cc:330] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb: stopping tablet replica
I20260926 02:11:56.183401 4820 raft_consensus.cc:2229] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:11:56.183439 4819 raft_consensus.cc:2229] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:11:56.184083 4820 raft_consensus.cc:2258] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:11:56.184180 4819 raft_consensus.cc:2258] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:11:56.185901 4821 tablet_replica.cc:330] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42: stopping tablet replica
I20260926 02:11:56.186492 4821 raft_consensus.cc:2229] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:11:56.186992 4819 ts_tablet_manager.cc:1905] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:11:56.187369 4821 raft_consensus.cc:2258] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:11:56.190804 4820 ts_tablet_manager.cc:1905] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:11:56.192881 4821 ts_tablet_manager.cc:1905] T cdcb885c490f45258325013bee177630 P 74232a70573049c6b7e29552e3394b42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:11:56.194655 4819 ts_tablet_manager.cc:1918] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:11:56.194870 4819 log.cc:1196] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/cdcb885c490f45258325013bee177630
I20260926 02:11:56.195498 4819 ts_tablet_manager.cc:1939] T cdcb885c490f45258325013bee177630 P f1a91475f17740ee85eaa5f25da6a617: Deleting consensus metadata
I20260926 02:11:56.197634 4308 catalog_manager.cc:4841] TS f1a91475f17740ee85eaa5f25da6a617 (127.18.159.130:45397): tablet cdcb885c490f45258325013bee177630 (table test-table [id=3e3e137b869f451ea6d092a9cf691fe9]) successfully deleted
I20260926 02:11:56.198858 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:56.175620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33566 (local address 127.18.159.129:36431)
0926 02:11:56.175867 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:56.175877 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:56.185145 (+ 9268us) server_negotiation.cc:407] Connection header received
0926 02:11:56.185399 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:56.185405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:56.185536 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:56.185669 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:56.186485 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:56.187967 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:56.190004 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:56.190669 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:56.190802 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:56.196806 (+ 6004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:56.196820 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:56.196839 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:11:56.197831 (+ 992us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:56.197896 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:11:56.197926 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:56.198031 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:56.198314 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 02:11:56.198439 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:11:56.199640 4820 ts_tablet_manager.cc:1918] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:11:56.199851 4820 log.cc:1196] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/cdcb885c490f45258325013bee177630
I20260926 02:11:56.199945 4508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:11:56.200603 4820 ts_tablet_manager.cc:1939] T cdcb885c490f45258325013bee177630 P 5bf139896e7d4f3b8e7648413e5a0fdb: Deleting consensus metadata
I20260926 02:11:56.203146 4310 catalog_manager.cc:4841] TS 5bf139896e7d4f3b8e7648413e5a0fdb (127.18.159.131:37997): tablet cdcb885c490f45258325013bee177630 (table test-table [id=3e3e137b869f451ea6d092a9cf691fe9]) successfully deleted
I20260926 02:11:56.298364 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 4385
W20260926 02:11:56.316686 4309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.159.129:36431 (error 108)
I20260926 02:11:56.317042 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 4517
I20260926 02:11:56.318889 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:56.318417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:36431 (local address 127.0.0.1:33580)
0926 02:11:56.318583 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:56.318718 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 02:11:56.319145 4309 catalog_manager.cc:4584] TS 74232a70573049c6b7e29552e3394b42 (127.18.159.129:36431): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cdcb885c490f45258325013bee177630: Network error: Client connection negotiation failed: client connection to 127.18.159.129:36431: connect: Connection refused (error 111)
I20260926 02:11:56.324384 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:56.318124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56688 (local address 127.18.159.130:45397)
0926 02:11:56.318319 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:56.318325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:56.318367 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:11:56.318438 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:56.318446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:56.318573 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:56.318670 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:56.319192 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:56.320028 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:56.321406 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:56.321829 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:56.321966 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:56.322998 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:56.323007 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:56.323021 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:11:56.323587 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:56.323639 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:11:56.323657 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:56.323781 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:56.323917 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 02:11:56.324027 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:11:56.325110 4640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:11:56.373445 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:56.372892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:36431 (local address 127.0.0.1:33590)
0926 02:11:56.373123 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:56.373233 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 02:11:56.375380 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:56.374571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:36431 (local address 127.0.0.1:33594)
0926 02:11:56.374809 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:56.375185 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260926 02:11:56.435946 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:56.435491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:36431 (local address 127.0.0.1:33606)
0926 02:11:56.435704 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:56.435788 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 02:11:56.436968 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:56.436637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:36431 (local address 127.0.0.1:33616)
0926 02:11:56.436787 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:56.436858 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 02:11:56.449738 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 4517
I20260926 02:11:56.468880 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 4648
I20260926 02:11:56.477246 4794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:56.469674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54152 (local address 127.18.159.131:37997)
0926 02:11:56.469917 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:56.469923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:56.469945 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:11:56.470116 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:56.470123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:56.470225 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:56.470348 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:56.470831 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:56.471608 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:56.473172 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:56.473620 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:56.473773 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:56.475180 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:56.475193 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:56.475213 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:11:56.476285 (+ 1072us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:56.476360 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:11:56.476392 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:56.476524 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:56.476704 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 02:11:56.476823 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:11:56.478025 4772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:11:56.517822 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:56.517332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:36431 (local address 127.0.0.1:33624)
0926 02:11:56.517544 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:56.517645 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 02:11:56.518908 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:56.518524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:36431 (local address 127.0.0.1:33634)
0926 02:11:56.518723 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:56.518793 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 02:11:56.575640 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 4648
I20260926 02:11:56.591851 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 4293
I20260926 02:11:56.592962 4356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:11:56.703553 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:56.702558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:36431 (local address 127.0.0.1:33650)
0926 02:11:56.703044 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:56.703203 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208}
I20260926 02:11:56.703791 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 4293
2026-09-26T02:11:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2959 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:11:56 dist-test-slave-03rn krb5kdc[4830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:11:56 dist-test-slave-03rn krb5kdc[4830](info): set up 2 sockets
Sep 26 02:11:56 dist-test-slave-03rn krb5kdc[4830](info): commencing operation
krb5kdc: starting...
W20260926 02:11:57.824731 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:11:57 dist-test-slave-03rn krb5kdc[4830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:11:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:11:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:11:58.080566 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:39813
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:45025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:39813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:11:58.425915 4843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:11:58.426278 4843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:11:58.426429 4843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:11:58.434041 4843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:11:58.434207 4843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:11:58.434326 4843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:11:58.434425 4843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:11:58.434512 4843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:11:58.444834 4843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:45025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:39813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:39813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:11:58.446256 4843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:11:58.447645 4843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:11:58.460228 4848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:58.460137 4849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:58.460278 4851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:11:58.461278 4843 server_base.cc:1034] running on GCE node
I20260926 02:11:58.462138 4843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:11:58.463646 4843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:11:58.464891 4843 hybrid_clock.cc:648] HybridClock initialized: now 1790388718464806 us; error 76 us; skew 500 ppm
I20260926 02:11:58.465289 4843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:11:58 dist-test-slave-03rn krb5kdc[4830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388718, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:11:58.468672 4843 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:11:58.470341 4843 webserver.cc:460] Webserver started at http://127.18.159.190:37961/ using document root <none> and password file <none>
I20260926 02:11:58.471091 4843 fs_manager.cc:359] Metadata directory not provided
I20260926 02:11:58.471228 4843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:11:58.471580 4843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:11:58.474888 4843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "ca710d2265c3477fa5b1407a96d4f8a8"
format_stamp: "Formatted at 2026-09-26 02:11:58 on dist-test-slave-03rn"
server_key: "621ffecf23b3bc69badf5019b63565d3"
server_key_iv: "2a1041ac32b0100a907d621fe5a42623"
server_key_version: "encryptionkey@0"
I20260926 02:11:58.475718 4843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "ca710d2265c3477fa5b1407a96d4f8a8"
format_stamp: "Formatted at 2026-09-26 02:11:58 on dist-test-slave-03rn"
server_key: "621ffecf23b3bc69badf5019b63565d3"
server_key_iv: "2a1041ac32b0100a907d621fe5a42623"
server_key_version: "encryptionkey@0"
I20260926 02:11:58.481444 4843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 02:11:58.486179 4858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:58.487895 4843 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:11:58.488093 4843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "ca710d2265c3477fa5b1407a96d4f8a8"
format_stamp: "Formatted at 2026-09-26 02:11:58 on dist-test-slave-03rn"
server_key: "621ffecf23b3bc69badf5019b63565d3"
server_key_iv: "2a1041ac32b0100a907d621fe5a42623"
server_key_version: "encryptionkey@0"
I20260926 02:11:58.488354 4843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:11:58.514650 4843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:11:58.519413 4843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:11:58.519671 4843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:11:58.539346 4843 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:39813
I20260926 02:11:58.539354 4909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:39813 every 8 connection(s)
I20260926 02:11:58.540869 4843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:11:58.544962 4910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:11:58.545827 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 4843
I20260926 02:11:58.546036 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:11:58.546577 19070 external_mini_cluster.cc:1428] Setting key 4835d4e50999964390f57a339c1f4ff9
I20260926 02:11:58.560073 4910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8: Bootstrap starting.
Sep 26 02:11:58 dist-test-slave-03rn krb5kdc[4830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:11:58.564262 4910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8: Neither blocks nor log segments found. Creating new log.
I20260926 02:11:58.565599 4910 log.cc:824] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8: Log is configured to *not* fsync() on all Append() calls
I20260926 02:11:58.569213 4910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8: No bootstrap required, opened a new log
I20260926 02:11:58.571359 4913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:58.548617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40380 (local address 127.18.159.190:39813)
0926 02:11:58.549432 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:58.549444 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:58.549486 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:11:58.551728 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:58.551754 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:58.552303 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:58.552679 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:58.554560 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:58.555899 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:58.557857 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:58.558594 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:58.558728 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:58.563760 (+ 5032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:58.563770 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:58.563786 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:58.563833 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:58.567107 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:58.568007 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:58.568017 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:58.568024 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:58.568249 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:58.568653 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:58.568656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:58.568658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:58.569178 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:58.569462 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:58.569922 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 02:11:58.570296 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":398,"threads_started":1}
I20260926 02:11:58.577553 4910 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca710d2265c3477fa5b1407a96d4f8a8" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39813 } }
I20260926 02:11:58.577960 4910 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:11:58.578085 4910 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca710d2265c3477fa5b1407a96d4f8a8, State: Initialized, Role: FOLLOWER
I20260926 02:11:58.578734 4910 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca710d2265c3477fa5b1407a96d4f8a8" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39813 } }
I20260926 02:11:58.578979 4910 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:11:58.579128 4910 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:11:58.579344 4910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:11:58.581740 4910 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca710d2265c3477fa5b1407a96d4f8a8" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39813 } }
I20260926 02:11:58.582296 4910 leader_election.cc:302] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca710d2265c3477fa5b1407a96d4f8a8; no voters:
I20260926 02:11:58.582927 4910 leader_election.cc:288] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:11:58.583089 4915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:11:58.584633 4915 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8 [term 1 LEADER]: Becoming Leader. State: Replica: ca710d2265c3477fa5b1407a96d4f8a8, State: Running, Role: LEADER
I20260926 02:11:58.585218 4910 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:11:58.585233 4915 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca710d2265c3477fa5b1407a96d4f8a8" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39813 } }
I20260926 02:11:58.588382 4917 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca710d2265c3477fa5b1407a96d4f8a8. Latest consensus state: current_term: 1 leader_uuid: "ca710d2265c3477fa5b1407a96d4f8a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca710d2265c3477fa5b1407a96d4f8a8" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39813 } } }
I20260926 02:11:58.588894 4917 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8 [sys.catalog]: This master's current role is: LEADER
I20260926 02:11:58.588693 4916 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca710d2265c3477fa5b1407a96d4f8a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca710d2265c3477fa5b1407a96d4f8a8" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39813 } } }
I20260926 02:11:58.589022 4916 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8 [sys.catalog]: This master's current role is: LEADER
I20260926 02:11:58.590912 4922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:11:58.596925 4922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:11:58.606730 4922 catalog_manager.cc:1347] Generated new cluster ID: 104a64b8becf4ab582b1fdeb3f3ce311
I20260926 02:11:58.607002 4922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:11:58.616284 4922 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:11:58.618310 4922 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:11:58.629765 4922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8: Generated new TSK 0
I20260926 02:11:58.630971 4922 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.159.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
I20260926 02:11:58.704748 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:39813
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:45025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:11:59.062105 4936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:11:59.062491 4936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:11:59.062669 4936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:11:59.070370 4936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:11:59.070564 4936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:11:59.070747 4936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:11:59.081499 4936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:45025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:39813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:11:59.082917 4936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:11:59.084348 4936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:11:59.099267 4941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:59.099188 4942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:59.099301 4944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:11:59.100209 4936 server_base.cc:1034] running on GCE node
I20260926 02:11:59.100943 4936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:11:59.101958 4936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:11:59.103171 4936 hybrid_clock.cc:648] HybridClock initialized: now 1790388719103117 us; error 39 us; skew 500 ppm
I20260926 02:11:59.103480 4936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:11:59 dist-test-slave-03rn krb5kdc[4830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388719, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:11:59.106797 4936 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:11:59.108498 4936 webserver.cc:460] Webserver started at http://127.18.159.129:44557/ using document root <none> and password file <none>
I20260926 02:11:59.109220 4936 fs_manager.cc:359] Metadata directory not provided
I20260926 02:11:59.109332 4936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:11:59.109625 4936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:11:59.112805 4936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "9f131f83858c48aba71713b6e1a854ee"
format_stamp: "Formatted at 2026-09-26 02:11:59 on dist-test-slave-03rn"
server_key: "f13350e82fdb76fb16dca25af99206da"
server_key_iv: "b4ecf5f2b8aa4dd31a6a28800dd00fad"
server_key_version: "encryptionkey@0"
I20260926 02:11:59.113600 4936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "9f131f83858c48aba71713b6e1a854ee"
format_stamp: "Formatted at 2026-09-26 02:11:59 on dist-test-slave-03rn"
server_key: "f13350e82fdb76fb16dca25af99206da"
server_key_iv: "b4ecf5f2b8aa4dd31a6a28800dd00fad"
server_key_version: "encryptionkey@0"
I20260926 02:11:59.119375 4936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:11:59.123318 4951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:59.124956 4936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:11:59.125123 4936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "9f131f83858c48aba71713b6e1a854ee"
format_stamp: "Formatted at 2026-09-26 02:11:59 on dist-test-slave-03rn"
server_key: "f13350e82fdb76fb16dca25af99206da"
server_key_iv: "b4ecf5f2b8aa4dd31a6a28800dd00fad"
server_key_version: "encryptionkey@0"
I20260926 02:11:59.125319 4936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:11:59.144371 4936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:11:59.148936 4936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:11:59.149199 4936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:11:59.150163 4936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:11:59.151487 4936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:11:59.151563 4936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:59.151650 4936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:11:59.151695 4936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:59.186565 4936 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:39611
I20260926 02:11:59.186620 5063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:39611 every 8 connection(s)
I20260926 02:11:59.187959 4936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:11:59.194615 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 4936
I20260926 02:11:59.194824 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:11:59.195251 19070 external_mini_cluster.cc:1428] Setting key db197ac205f15cd13cf68870d3b82cf0
Sep 26 02:11:59 dist-test-slave-03rn krb5kdc[4830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388719, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:11:59.212926 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:59.190739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:38349 (local address 127.18.159.190:39813)
0926 02:11:59.191280 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:59.191284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:59.192416 (+ 1132us) server_negotiation.cc:407] Connection header received
0926 02:11:59.194506 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:59.194512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:59.194609 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:59.194731 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:59.196551 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:59.197445 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:59.200205 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:59.200651 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:59.200755 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:59.205510 (+ 4755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:59.205516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:59.205519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:59.205556 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:59.209132 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:59.210035 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:59.210044 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:59.210050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:59.210226 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:59.210702 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:59.210710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:59.210715 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:59.211133 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:59.211311 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:59.212095 (+ 784us) server_negotiation.cc:299] Negotiation successful
0926 02:11:59.212415 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":276,"threads_started":1}
I20260926 02:11:59.213647 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:59.191397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:39813 (local address 127.18.159.129:38349)
0926 02:11:59.192213 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:59.192267 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:59.194199 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:59.195010 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:59.195019 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:59.195624 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:11:59.196347 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:59.196377 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:59.197625 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:59.197631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:59.199980 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:59.200002 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:59.200924 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:59.200933 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:59.201287 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:59.202066 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:11:59.202107 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:11:59.205213 (+ 3106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:11:59.209339 (+ 4126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:11:59.209348 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:11:59.209364 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:11:59.209860 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:11:59.210371 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:11:59.210374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:11:59.210376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:11:59.210570 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:11:59.211276 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:11:59.211282 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:11:59.211674 (+ 392us) client_negotiation.cc:769] Sending connection context
0926 02:11:59.211928 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 02:11:59.212448 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":275,"threads_started":1}
I20260926 02:11:59.215349 5064 heartbeater.cc:347] Connected to a master server at 127.18.159.190:39813
I20260926 02:11:59.215700 5064 heartbeater.cc:464] Registering TS with master...
I20260926 02:11:59.216699 5064 heartbeater.cc:511] Master 127.18.159.190:39813 requested a full tablet report, sending...
I20260926 02:11:59.219087 4875 ts_manager.cc:194] Registered new tserver with Master: 9f131f83858c48aba71713b6e1a854ee (127.18.159.129:39611)
I20260926 02:11:59.220804 4875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:38349
WARNING: no policy specified for kudu/127.18.159.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
I20260926 02:11:59.256233 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:39813
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:45025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:11:59.600216 5070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:11:59.600549 5070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:11:59.600651 5070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:11:59.608004 5070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:11:59.608141 5070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:11:59.608348 5070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:11:59.618366 5070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:45025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:39813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:11:59.619683 5070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:11:59.621098 5070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:11:59.635042 5078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:59.635100 5075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:11:59.635154 5076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:11:59.641579 5070 server_base.cc:1034] running on GCE node
I20260926 02:11:59.642290 5070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:11:59.643348 5070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:11:59.644594 5070 hybrid_clock.cc:648] HybridClock initialized: now 1790388719644544 us; error 55 us; skew 500 ppm
I20260926 02:11:59.644932 5070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:11:59 dist-test-slave-03rn krb5kdc[4830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388719, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:11:59.648540 5070 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:11:59.650254 5070 webserver.cc:460] Webserver started at http://127.18.159.130:35743/ using document root <none> and password file <none>
I20260926 02:11:59.650990 5070 fs_manager.cc:359] Metadata directory not provided
I20260926 02:11:59.651168 5070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:11:59.651460 5070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:11:59.654587 5070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "1890c7de3b05421b94c9ccad9158d325"
format_stamp: "Formatted at 2026-09-26 02:11:59 on dist-test-slave-03rn"
server_key: "fa620a6d22d0f7147a91de1fecc74ee1"
server_key_iv: "e9bc9d1d44bbc03aa576d1b471edd414"
server_key_version: "encryptionkey@0"
I20260926 02:11:59.655367 5070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "1890c7de3b05421b94c9ccad9158d325"
format_stamp: "Formatted at 2026-09-26 02:11:59 on dist-test-slave-03rn"
server_key: "fa620a6d22d0f7147a91de1fecc74ee1"
server_key_iv: "e9bc9d1d44bbc03aa576d1b471edd414"
server_key_version: "encryptionkey@0"
I20260926 02:11:59.661135 5070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260926 02:11:59.665282 5085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:59.667095 5070 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:11:59.667341 5070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "1890c7de3b05421b94c9ccad9158d325"
format_stamp: "Formatted at 2026-09-26 02:11:59 on dist-test-slave-03rn"
server_key: "fa620a6d22d0f7147a91de1fecc74ee1"
server_key_iv: "e9bc9d1d44bbc03aa576d1b471edd414"
server_key_version: "encryptionkey@0"
I20260926 02:11:59.667595 5070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:11:59.691687 5070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:11:59.696694 5070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:11:59.696959 5070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:11:59.697973 5070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:11:59.699481 5070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:11:59.699561 5070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:59.699652 5070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:11:59.699697 5070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:11:59.734877 5070 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:40149
I20260926 02:11:59.734973 5198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:40149 every 8 connection(s)
I20260926 02:11:59.736315 5070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:11:59.743255 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 5070
I20260926 02:11:59.743440 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:11:59.743880 19070 external_mini_cluster.cc:1428] Setting key d048204708fadd3e50bbf435c6ed64cb
Sep 26 02:11:59 dist-test-slave-03rn krb5kdc[4830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388719, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:11:59.759927 5201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:59.739094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:42207 (local address 127.18.159.190:39813)
0926 02:11:59.739592 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0926 02:11:59.739597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:11:59.740651 (+ 1054us) server_negotiation.cc:407] Connection header received
0926 02:11:59.742701 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:11:59.742705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:11:59.742780 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:11:59.742959 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:11:59.745002 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:59.745908 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:59.747582 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:11:59.748410 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:11:59.748563 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:59.752590 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:11:59.752599 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:11:59.752607 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:11:59.752662 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:11:59.755908 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:59.757209 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:59.757214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:59.757216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:59.757392 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:11:59.757976 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:11:59.757979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:11:59.757982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:11:59.758283 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:11:59.758419 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 02:11:59.759436 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0926 02:11:59.759642 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":266,"threads_started":1}
I20260926 02:11:59.761514 5202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:11:59.739698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:39813 (local address 127.18.159.130:42207)
0926 02:11:59.740484 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0926 02:11:59.740524 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 02:11:59.742376 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:11:59.743233 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:11:59.743242 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:11:59.743948 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:11:59.744806 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:59.744830 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:59.746094 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:59.746098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:59.747416 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:11:59.747429 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:11:59.748632 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:11:59.748637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:11:59.748848 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:11:59.749588 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:11:59.749612 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:11:59.752284 (+ 2672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:11:59.756236 (+ 3952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:11:59.756251 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:11:59.756276 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:11:59.757022 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:11:59.757527 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:11:59.757535 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:11:59.757540 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:11:59.757835 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:11:59.758435 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:11:59.758447 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:11:59.758971 (+ 524us) client_negotiation.cc:769] Sending connection context
0926 02:11:59.759304 (+ 333us) client_negotiation.cc:240] Negotiation successful
0926 02:11:59.759927 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":277,"threads_started":1}
I20260926 02:11:59.763238 5199 heartbeater.cc:347] Connected to a master server at 127.18.159.190:39813
I20260926 02:11:59.763599 5199 heartbeater.cc:464] Registering TS with master...
I20260926 02:11:59.764526 5199 heartbeater.cc:511] Master 127.18.159.190:39813 requested a full tablet report, sending...
I20260926 02:11:59.766202 4875 ts_manager.cc:194] Registered new tserver with Master: 1890c7de3b05421b94c9ccad9158d325 (127.18.159.130:40149)
I20260926 02:11:59.767110 4875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:42207
WARNING: no policy specified for kudu/127.18.159.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
I20260926 02:11:59.803745 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:39813
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:45025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:00.149231 5205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:00.149585 5205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:00.149688 5205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:00.157217 5205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:00.157361 5205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:00.157541 5205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:12:00.167848 5205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:45025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:39813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:00.169440 5205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:00.170892 5205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:00.185907 5211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:00.185883 5210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:00.187052 5213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:00.189524 5205 server_base.cc:1034] running on GCE node
I20260926 02:12:00.190227 5205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:00.191402 5205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:00.192623 5205 hybrid_clock.cc:648] HybridClock initialized: now 1790388720192545 us; error 84 us; skew 500 ppm
I20260926 02:12:00.193014 5205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:00 dist-test-slave-03rn krb5kdc[4830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388720, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:00.196359 5205 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:12:00.197904 5205 webserver.cc:460] Webserver started at http://127.18.159.131:46459/ using document root <none> and password file <none>
I20260926 02:12:00.198647 5205 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:00.198793 5205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:00.199116 5205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:00.202311 5205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "b44acfa7d5254590b4fdf34f15195d47"
format_stamp: "Formatted at 2026-09-26 02:12:00 on dist-test-slave-03rn"
server_key: "7fd0a1cbcae7d1f6478e8c18e6c5f829"
server_key_iv: "6a15d97faa7a9316cbb875f4f7f3b98c"
server_key_version: "encryptionkey@0"
I20260926 02:12:00.203188 5205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "b44acfa7d5254590b4fdf34f15195d47"
format_stamp: "Formatted at 2026-09-26 02:12:00 on dist-test-slave-03rn"
server_key: "7fd0a1cbcae7d1f6478e8c18e6c5f829"
server_key_iv: "6a15d97faa7a9316cbb875f4f7f3b98c"
server_key_version: "encryptionkey@0"
I20260926 02:12:00.209043 5205 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:12:00.213219 5220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:00.214847 5205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260926 02:12:00.215070 5205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "b44acfa7d5254590b4fdf34f15195d47"
format_stamp: "Formatted at 2026-09-26 02:12:00 on dist-test-slave-03rn"
server_key: "7fd0a1cbcae7d1f6478e8c18e6c5f829"
server_key_iv: "6a15d97faa7a9316cbb875f4f7f3b98c"
server_key_version: "encryptionkey@0"
I20260926 02:12:00.215315 5205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:00.224900 5064 heartbeater.cc:503] Master 127.18.159.190:39813 was elected leader, sending a full tablet report...
I20260926 02:12:00.248675 5205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:00.253760 5205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:00.254096 5205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:00.255167 5205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:12:00.256775 5205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:12:00.256884 5205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:00.257050 5205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:12:00.257123 5205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:00.293473 5205 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:35651
I20260926 02:12:00.293507 5333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:35651 every 8 connection(s)
I20260926 02:12:00.294960 5205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:12:00.300606 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 5205
I20260926 02:12:00.300799 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:12:00.301251 19070 external_mini_cluster.cc:1428] Setting key 55fa8be1e0cdfbdc6da4a632ccefd203
Sep 26 02:12:00 dist-test-slave-03rn krb5kdc[4830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388720, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:12:00.318150 5336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:00.297830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:42503 (local address 127.18.159.190:39813)
0926 02:12:00.298305 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:00.298312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:00.299561 (+ 1249us) server_negotiation.cc:407] Connection header received
0926 02:12:00.301966 (+ 2405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:00.301971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:00.302047 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:00.302141 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:00.304134 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.305120 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.306936 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.307349 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.307452 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:00.311555 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:00.311561 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:00.311564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:00.311603 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:00.314441 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:00.315455 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:00.315464 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:00.315470 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:00.315646 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:00.316232 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:00.316236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:00.316238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:00.316625 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:00.316746 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:00.317677 (+ 931us) server_negotiation.cc:299] Negotiation successful
0926 02:12:00.317876 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":219,"threads_started":1}
I20260926 02:12:00.319216 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:00.298567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:39813 (local address 127.18.159.131:42503)
0926 02:12:00.299362 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:00.299410 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:00.301601 (+ 2191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:00.302475 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:00.302491 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:00.303137 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:00.303945 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:00.303967 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.305311 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.305318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:00.306739 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:00.306754 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.307585 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.307590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:00.307846 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:00.308740 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:00.308774 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:00.311281 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:00.314619 (+ 3338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:00.314627 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:00.314643 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:00.315310 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:00.315775 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:00.315780 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:00.315787 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:00.316081 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:00.316750 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:00.316758 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:00.317217 (+ 459us) client_negotiation.cc:769] Sending connection context
0926 02:12:00.317553 (+ 336us) client_negotiation.cc:240] Negotiation successful
0926 02:12:00.317968 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":314,"threads_started":1}
I20260926 02:12:00.320888 5334 heartbeater.cc:347] Connected to a master server at 127.18.159.190:39813
I20260926 02:12:00.321249 5334 heartbeater.cc:464] Registering TS with master...
I20260926 02:12:00.322041 5334 heartbeater.cc:511] Master 127.18.159.190:39813 requested a full tablet report, sending...
I20260926 02:12:00.323592 4875 ts_manager.cc:194] Registered new tserver with Master: b44acfa7d5254590b4fdf34f15195d47 (127.18.159.131:35651)
I20260926 02:12:00.324514 4875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:42503
I20260926 02:12:00.330313 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:12:00.349618 5336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:00.335016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60526 (local address 127.18.159.190:39813)
0926 02:12:00.335223 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:00.335229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:00.335726 (+ 497us) server_negotiation.cc:407] Connection header received
0926 02:12:00.335921 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:00.335925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:00.335995 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:00.336091 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:00.337824 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.338590 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.340253 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.340724 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.340863 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:00.343670 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:00.343675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:00.343679 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:00.343717 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:00.346588 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:00.347422 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:00.347426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:00.347428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:00.347552 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:00.348068 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:00.348072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:00.348074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:00.348553 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:00.348805 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:00.349178 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 02:12:00.349373 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:12:00.353987 4875 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:12:00.361202 4875 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:12:00.407464 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:00.384829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42422 (local address 127.18.159.130:40149)
0926 02:12:00.398105 (+ 13276us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:00.398113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:00.398136 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:12:00.398230 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:00.398239 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:00.398539 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:00.398784 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:00.400123 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.401547 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.404891 (+ 3344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.406090 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.406297 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:00.406522 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:00.406923 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 02:12:00.407115 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13123,"thread_start_us":13171,"threads_started":1}
I20260926 02:12:00.407578 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:00.384072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:40149 (local address 127.0.0.1:42422)
0926 02:12:00.397525 (+ 13453us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:00.397548 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:00.397660 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:00.399095 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:00.399104 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:00.399149 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:00.399909 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:00.399924 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.402024 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.402030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:00.404691 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:00.404707 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.406359 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.406366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:00.406620 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:00.406702 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 02:12:00.406875 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 02:12:00.407091 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13330,"mutex_wait_us":47,"spinlock_wait_cycles":11520,"thread_start_us":12340,"threads_started":1}
I20260926 02:12:00.409188 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:00.380472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:39611 (local address 127.0.0.1:38840)
0926 02:12:00.393734 (+ 13262us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:00.393811 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:00.394095 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:00.399095 (+ 5000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:00.399104 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:00.399156 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:00.399995 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:00.400032 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.402031 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.402040 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:00.405641 (+ 3601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:00.405654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.408327 (+ 2673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.408332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:00.408459 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:00.408482 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:12:00.408586 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 02:12:00.408729 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12874,"mutex_wait_us":100,"spinlock_wait_cycles":4608,"thread_start_us":12661,"threads_started":1}
I20260926 02:12:00.409206 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:00.380616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38840 (local address 127.18.159.129:39611)
0926 02:12:00.398074 (+ 17458us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:00.398085 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:00.398119 (+ 34us) server_negotiation.cc:407] Connection header received
0926 02:12:00.398232 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:00.398239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:00.398610 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:00.398784 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:00.400290 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.401855 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.405850 (+ 3995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.407411 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.407624 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:00.407803 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:00.408657 (+ 854us) server_negotiation.cc:299] Negotiation successful
0926 02:12:00.408848 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17253,"thread_start_us":17022,"threads_started":1}
I20260926 02:12:00.412253 5133 tablet_service.cc:1467] Processing CreateTablet for tablet 6351ff0e950245408237a77ab618ffed (DEFAULT_TABLE table=test-table [id=5294400afd6747338aff0c4965900151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:12:00.413240 4999 tablet_service.cc:1467] Processing CreateTablet for tablet 6351ff0e950245408237a77ab618ffed (DEFAULT_TABLE table=test-table [id=5294400afd6747338aff0c4965900151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:12:00.414099 5351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:00.380927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:35651 (local address 127.0.0.1:37352)
0926 02:12:00.396771 (+ 15844us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:00.396801 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:00.396991 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:00.400485 (+ 3494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:00.400490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:00.400524 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:00.400863 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:00.400874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.402669 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.402677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:00.405638 (+ 2961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:00.405654 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.413140 (+ 7486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.413148 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:00.413340 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:00.413385 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 02:12:00.413488 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 02:12:00.413639 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15661,"mutex_wait_us":26,"thread_start_us":16339,"threads_started":1}
I20260926 02:12:00.414177 5133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6351ff0e950245408237a77ab618ffed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:00.414784 5350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:00.381299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37352 (local address 127.18.159.131:35651)
0926 02:12:00.399730 (+ 18431us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:00.399742 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:00.399781 (+ 39us) server_negotiation.cc:407] Connection header received
0926 02:12:00.399895 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:00.399904 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:00.400258 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:00.400485 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:00.401049 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.402528 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.407935 (+ 5407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.412895 (+ 4960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.413140 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:00.413323 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:00.414355 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0926 02:12:00.414536 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18212,"thread_start_us":18054,"threads_started":1}
I20260926 02:12:00.415175 4999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6351ff0e950245408237a77ab618ffed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:00.418202 5269 tablet_service.cc:1467] Processing CreateTablet for tablet 6351ff0e950245408237a77ab618ffed (DEFAULT_TABLE table=test-table [id=5294400afd6747338aff0c4965900151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:12:00.419700 5269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6351ff0e950245408237a77ab618ffed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:00.430704 5354 tablet_bootstrap.cc:492] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee: Bootstrap starting.
I20260926 02:12:00.433519 5355 tablet_bootstrap.cc:492] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47: Bootstrap starting.
I20260926 02:12:00.434957 5354 tablet_bootstrap.cc:654] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:00.436079 5354 log.cc:824] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:00.437433 5353 tablet_bootstrap.cc:492] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325: Bootstrap starting.
I20260926 02:12:00.438477 5355 tablet_bootstrap.cc:654] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:00.439605 5355 log.cc:824] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:00.441056 5353 tablet_bootstrap.cc:654] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:00.442365 5353 log.cc:824] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:00.442965 5355 tablet_bootstrap.cc:492] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47: No bootstrap required, opened a new log
I20260926 02:12:00.443042 5354 tablet_bootstrap.cc:492] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee: No bootstrap required, opened a new log
I20260926 02:12:00.443352 5354 ts_tablet_manager.cc:1397] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260926 02:12:00.443373 5355 ts_tablet_manager.cc:1397] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260926 02:12:00.445859 5353 tablet_bootstrap.cc:492] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325: No bootstrap required, opened a new log
I20260926 02:12:00.446177 5353 ts_tablet_manager.cc:1397] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 02:12:00.451699 5355 raft_consensus.cc:348] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f131f83858c48aba71713b6e1a854ee" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39611 } } peers { permanent_uuid: "b44acfa7d5254590b4fdf34f15195d47" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35651 } } peers { permanent_uuid: "1890c7de3b05421b94c9ccad9158d325" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 40149 } }
I20260926 02:12:00.452121 5355 raft_consensus.cc:374] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:00.452320 5355 raft_consensus.cc:729] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44acfa7d5254590b4fdf34f15195d47, State: Initialized, Role: FOLLOWER
I20260926 02:12:00.452989 5355 consensus_queue.cc:260] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f131f83858c48aba71713b6e1a854ee" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39611 } } peers { permanent_uuid: "b44acfa7d5254590b4fdf34f15195d47" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35651 } } peers { permanent_uuid: "1890c7de3b05421b94c9ccad9158d325" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 40149 } }
I20260926 02:12:00.454236 5334 heartbeater.cc:503] Master 127.18.159.190:39813 was elected leader, sending a full tablet report...
I20260926 02:12:00.454766 5355 ts_tablet_manager.cc:1428] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260926 02:12:00.454593 5354 raft_consensus.cc:348] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f131f83858c48aba71713b6e1a854ee" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39611 } } peers { permanent_uuid: "b44acfa7d5254590b4fdf34f15195d47" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35651 } } peers { permanent_uuid: "1890c7de3b05421b94c9ccad9158d325" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 40149 } }
I20260926 02:12:00.455061 5354 raft_consensus.cc:374] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:00.455169 5354 raft_consensus.cc:729] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f131f83858c48aba71713b6e1a854ee, State: Initialized, Role: FOLLOWER
I20260926 02:12:00.455808 5354 consensus_queue.cc:260] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f131f83858c48aba71713b6e1a854ee" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39611 } } peers { permanent_uuid: "b44acfa7d5254590b4fdf34f15195d47" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35651 } } peers { permanent_uuid: "1890c7de3b05421b94c9ccad9158d325" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 40149 } }
I20260926 02:12:00.457716 5354 ts_tablet_manager.cc:1428] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260926 02:12:00.458591 5353 raft_consensus.cc:348] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f131f83858c48aba71713b6e1a854ee" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39611 } } peers { permanent_uuid: "b44acfa7d5254590b4fdf34f15195d47" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35651 } } peers { permanent_uuid: "1890c7de3b05421b94c9ccad9158d325" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 40149 } }
I20260926 02:12:00.459026 5353 raft_consensus.cc:374] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:00.459184 5353 raft_consensus.cc:729] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1890c7de3b05421b94c9ccad9158d325, State: Initialized, Role: FOLLOWER
I20260926 02:12:00.459767 5353 consensus_queue.cc:260] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f131f83858c48aba71713b6e1a854ee" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39611 } } peers { permanent_uuid: "b44acfa7d5254590b4fdf34f15195d47" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35651 } } peers { permanent_uuid: "1890c7de3b05421b94c9ccad9158d325" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 40149 } }
I20260926 02:12:00.460958 5199 heartbeater.cc:503] Master 127.18.159.190:39813 was elected leader, sending a full tablet report...
I20260926 02:12:00.461931 5353 ts_tablet_manager.cc:1428] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260926 02:12:00.488780 5200 tablet.cc:2369] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:12:00.519407 5361 raft_consensus.cc:482] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:12:00.519678 5361 raft_consensus.cc:504] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f131f83858c48aba71713b6e1a854ee" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39611 } } peers { permanent_uuid: "b44acfa7d5254590b4fdf34f15195d47" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35651 } } peers { permanent_uuid: "1890c7de3b05421b94c9ccad9158d325" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 40149 } }
I20260926 02:12:00.521576 5361 leader_election.cc:288] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f131f83858c48aba71713b6e1a854ee (127.18.159.129:39611), b44acfa7d5254590b4fdf34f15195d47 (127.18.159.131:35651)
I20260926 02:12:00.523303 5360 raft_consensus.cc:482] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:12:00.523761 5360 raft_consensus.cc:504] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f131f83858c48aba71713b6e1a854ee" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39611 } } peers { permanent_uuid: "b44acfa7d5254590b4fdf34f15195d47" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35651 } } peers { permanent_uuid: "1890c7de3b05421b94c9ccad9158d325" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 40149 } }
I20260926 02:12:00.532480 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:00.522040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:39615 (local address 127.18.159.129:39611)
0926 02:12:00.522350 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:00.522360 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:00.522868 (+ 508us) server_negotiation.cc:407] Connection header received
0926 02:12:00.523196 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:00.523203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:00.523312 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:00.523406 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:00.524408 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.525519 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.528916 (+ 3397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.530700 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.530881 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:00.530990 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:00.531933 (+ 943us) server_negotiation.cc:299] Negotiation successful
0926 02:12:00.532136 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 02:12:00.532850 5350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:00.522139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:55007 (local address 127.18.159.131:35651)
0926 02:12:00.522591 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:00.522599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:00.523358 (+ 759us) server_negotiation.cc:407] Connection header received
0926 02:12:00.523565 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:00.523571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:00.523753 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:00.523835 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:00.524977 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.525933 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.529285 (+ 3352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.531053 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.531232 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:00.531353 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:00.532208 (+ 855us) server_negotiation.cc:299] Negotiation successful
0926 02:12:00.532426 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254}
I20260926 02:12:00.533303 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:00.521929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:39611 (local address 127.18.159.130:39615)
0926 02:12:00.522753 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:00.522769 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:00.523014 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:00.523750 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:00.523757 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:00.523787 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:00.524240 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:00.524252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.525687 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.525695 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:00.528662 (+ 2967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:00.528683 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.531281 (+ 2598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.531290 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:00.531712 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:00.531751 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 02:12:00.532714 (+ 963us) client_negotiation.cc:240] Negotiation successful
0926 02:12:00.532874 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":1,"thread_start_us":462,"threads_started":1}
I20260926 02:12:00.534004 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:00.521950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:35651 (local address 127.18.159.130:55007)
0926 02:12:00.523174 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:00.523192 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:00.523397 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:00.524050 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:00.524059 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:00.524087 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:00.524804 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:00.524818 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.526079 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.526087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:00.529034 (+ 2947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:00.529056 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.531670 (+ 2614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.531680 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:00.531952 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:00.531999 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 02:12:00.533435 (+ 1436us) client_negotiation.cc:240] Negotiation successful
0926 02:12:00.533644 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"thread_start_us":857,"threads_started":1}
I20260926 02:12:00.534197 5360 leader_election.cc:288] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b44acfa7d5254590b4fdf34f15195d47 (127.18.159.131:35651), 1890c7de3b05421b94c9ccad9158d325 (127.18.159.130:40149)
I20260926 02:12:00.534597 5019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6351ff0e950245408237a77ab618ffed" candidate_uuid: "1890c7de3b05421b94c9ccad9158d325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f131f83858c48aba71713b6e1a854ee" is_pre_election: true
I20260926 02:12:00.535240 5019 raft_consensus.cc:2454] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1890c7de3b05421b94c9ccad9158d325 in term 0.
I20260926 02:12:00.536350 5086 leader_election.cc:302] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1890c7de3b05421b94c9ccad9158d325, 9f131f83858c48aba71713b6e1a854ee; no voters:
I20260926 02:12:00.536914 5361 raft_consensus.cc:2781] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:12:00.537073 5361 raft_consensus.cc:482] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:12:00.537178 5361 raft_consensus.cc:3037] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:00.539166 5289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6351ff0e950245408237a77ab618ffed" candidate_uuid: "1890c7de3b05421b94c9ccad9158d325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44acfa7d5254590b4fdf34f15195d47" is_pre_election: true
I20260926 02:12:00.539827 5289 raft_consensus.cc:2454] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1890c7de3b05421b94c9ccad9158d325 in term 0.
I20260926 02:12:00.540767 5361 raft_consensus.cc:504] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f131f83858c48aba71713b6e1a854ee" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39611 } } peers { permanent_uuid: "b44acfa7d5254590b4fdf34f15195d47" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35651 } } peers { permanent_uuid: "1890c7de3b05421b94c9ccad9158d325" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 40149 } }
I20260926 02:12:00.541635 5361 leader_election.cc:288] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 [CANDIDATE]: Term 1 election: Requested vote from peers 9f131f83858c48aba71713b6e1a854ee (127.18.159.129:39611), b44acfa7d5254590b4fdf34f15195d47 (127.18.159.131:35651)
I20260926 02:12:00.541941 5019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6351ff0e950245408237a77ab618ffed" candidate_uuid: "1890c7de3b05421b94c9ccad9158d325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f131f83858c48aba71713b6e1a854ee"
I20260926 02:12:00.542208 5289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6351ff0e950245408237a77ab618ffed" candidate_uuid: "1890c7de3b05421b94c9ccad9158d325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44acfa7d5254590b4fdf34f15195d47"
I20260926 02:12:00.542197 5019 raft_consensus.cc:3037] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:00.542404 5289 raft_consensus.cc:3037] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:00.545958 5019 raft_consensus.cc:2454] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1890c7de3b05421b94c9ccad9158d325 in term 1.
I20260926 02:12:00.545971 5289 raft_consensus.cc:2454] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1890c7de3b05421b94c9ccad9158d325 in term 1.
I20260926 02:12:00.546437 5087 leader_election.cc:302] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1890c7de3b05421b94c9ccad9158d325, b44acfa7d5254590b4fdf34f15195d47; no voters:
W20260926 02:12:00.546865 5335 tablet.cc:2369] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:12:00.547138 5361 raft_consensus.cc:2781] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:00.547495 5350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:00.526318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:33843 (local address 127.18.159.131:35651)
0926 02:12:00.533389 (+ 7071us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:00.533396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:00.537853 (+ 4457us) server_negotiation.cc:407] Connection header received
0926 02:12:00.538302 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:00.538307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:00.538397 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:00.538464 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:00.539373 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.540825 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.544902 (+ 4077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.546224 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.546355 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:00.546428 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:00.547062 (+ 634us) server_negotiation.cc:299] Negotiation successful
0926 02:12:00.547205 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6922,"thread_start_us":11888,"threads_started":1}
I20260926 02:12:00.548185 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:00.526142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:35651 (local address 127.18.159.129:33843)
0926 02:12:00.537724 (+ 11582us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:00.537752 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:00.538114 (+ 362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:00.538650 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:00.538654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:00.538676 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:00.539172 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:00.539188 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.541014 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.541019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:00.544016 (+ 2997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:00.544034 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.546646 (+ 2612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.546652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:00.546874 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:00.546915 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 02:12:00.547650 (+ 735us) client_negotiation.cc:240] Negotiation successful
0926 02:12:00.547806 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11426,"thread_start_us":11732,"threads_started":1}
I20260926 02:12:00.548523 5361 raft_consensus.cc:686] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 [term 1 LEADER]: Becoming Leader. State: Replica: 1890c7de3b05421b94c9ccad9158d325, State: Running, Role: LEADER
I20260926 02:12:00.549062 5289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6351ff0e950245408237a77ab618ffed" candidate_uuid: "9f131f83858c48aba71713b6e1a854ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44acfa7d5254590b4fdf34f15195d47" is_pre_election: true
I20260926 02:12:00.549399 5289 raft_consensus.cc:2379] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f131f83858c48aba71713b6e1a854ee in current term 1: Already voted for candidate 1890c7de3b05421b94c9ccad9158d325 in this term.
I20260926 02:12:00.549767 5361 consensus_queue.cc:237] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f131f83858c48aba71713b6e1a854ee" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39611 } } peers { permanent_uuid: "b44acfa7d5254590b4fdf34f15195d47" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35651 } } peers { permanent_uuid: "1890c7de3b05421b94c9ccad9158d325" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 40149 } }
I20260926 02:12:00.554263 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:00.532356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:40149 (local address 127.18.159.129:46853)
0926 02:12:00.537979 (+ 5623us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:00.537995 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:00.538493 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:00.541241 (+ 2748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:00.541246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:00.541279 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:00.544607 (+ 3328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:00.544619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.548743 (+ 4124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.548749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:00.551372 (+ 2623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:00.551389 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.553340 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.553350 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:00.553579 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:00.553631 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 02:12:00.553731 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 02:12:00.553893 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5500,"thread_start_us":5019,"threads_started":1}
I20260926 02:12:00.555302 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:00.532918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:46853 (local address 127.18.159.130:40149)
0926 02:12:00.533143 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:00.533152 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:00.538072 (+ 4920us) server_negotiation.cc:407] Connection header received
0926 02:12:00.538647 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:00.538652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:00.538730 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:00.538820 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:00.544789 (+ 5969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.545898 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.551553 (+ 5655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.553039 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.554799 (+ 1760us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:00.554882 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:00.554934 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 02:12:00.555065 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 02:12:00.556257 5154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6351ff0e950245408237a77ab618ffed" candidate_uuid: "9f131f83858c48aba71713b6e1a854ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1890c7de3b05421b94c9ccad9158d325" is_pre_election: true
I20260926 02:12:00.555879 4872 catalog_manager.cc:5495] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1890c7de3b05421b94c9ccad9158d325 (127.18.159.130). New cstate: current_term: 1 leader_uuid: "1890c7de3b05421b94c9ccad9158d325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f131f83858c48aba71713b6e1a854ee" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b44acfa7d5254590b4fdf34f15195d47" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1890c7de3b05421b94c9ccad9158d325" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 40149 } health_report { overall_health: HEALTHY } } }
I20260926 02:12:00.557366 4955 leader_election.cc:302] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f131f83858c48aba71713b6e1a854ee; no voters: 1890c7de3b05421b94c9ccad9158d325, b44acfa7d5254590b4fdf34f15195d47
I20260926 02:12:00.557893 5360 raft_consensus.cc:2726] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:12:00.605931 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:00.600367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.18.159.130:40149)
0926 02:12:00.600537 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:00.600542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:00.600659 (+ 117us) server_negotiation.cc:407] Connection header received
0926 02:12:00.600901 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:00.600905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:00.601006 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:00.601140 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:00.601622 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.602380 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.603777 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.604222 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.604330 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:00.604822 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:00.605348 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:00.605503 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:00.605642 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 02:12:00.605685 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:12:00.614282 5289 raft_consensus.cc:1261] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47 [term 1 FOLLOWER]: Refusing update from remote peer 1890c7de3b05421b94c9ccad9158d325: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:12:00.614506 5019 raft_consensus.cc:1261] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee [term 1 FOLLOWER]: Refusing update from remote peer 1890c7de3b05421b94c9ccad9158d325: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:12:00.615269 5367 consensus_queue.cc:1035] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b44acfa7d5254590b4fdf34f15195d47" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:12:00.615723 5361 consensus_queue.cc:1035] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f131f83858c48aba71713b6e1a854ee" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 39611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:12:00.630952 5372 mvcc.cc:205] Tried to move back new op lower bound from 7333432199625502720 to 7333432199377543168. Current Snapshot: MvccSnapshot[applied={T|T < 7333432199625502720}]
I20260926 02:12:00.643920 4873 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:12:00.644287 4873 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:12:00.648487 4873 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca710d2265c3477fa5b1407a96d4f8a8: Sending DeleteTablet for 3 replicas of tablet 6351ff0e950245408237a77ab618ffed
I20260926 02:12:00.649760 4999 tablet_service.cc:1514] Processing DeleteTablet for tablet 6351ff0e950245408237a77ab618ffed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:12:00 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:38840
I20260926 02:12:00.649808 5269 tablet_service.cc:1514] Processing DeleteTablet for tablet 6351ff0e950245408237a77ab618ffed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:12:00 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:37352
I20260926 02:12:00.650388 5133 tablet_service.cc:1514] Processing DeleteTablet for tablet 6351ff0e950245408237a77ab618ffed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:12:00 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:42422
I20260926 02:12:00.655331 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 4936
I20260926 02:12:00.659181 5382 tablet_replica.cc:330] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325: stopping tablet replica
I20260926 02:12:00.660481 5382 raft_consensus.cc:2229] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:12:00.662187 5382 raft_consensus.cc:2258] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:12:00.674295 5382 ts_tablet_manager.cc:1905] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:12:00.677862 5381 tablet_replica.cc:330] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee: stopping tablet replica
I20260926 02:12:00.678323 5381 raft_consensus.cc:2229] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:12:00.678818 5381 raft_consensus.cc:2258] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:12:00.681663 5381 ts_tablet_manager.cc:1905] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:12:00.688763 5382 ts_tablet_manager.cc:1918] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:12:00.689086 5382 log.cc:1196] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/6351ff0e950245408237a77ab618ffed
I20260926 02:12:00.689109 5381 ts_tablet_manager.cc:1918] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:12:00.689323 5381 log.cc:1196] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/6351ff0e950245408237a77ab618ffed
Sep 26 02:12:00 dist-test-slave-03rn krb5kdc[4830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:12:00.689885 5382 ts_tablet_manager.cc:1939] T 6351ff0e950245408237a77ab618ffed P 1890c7de3b05421b94c9ccad9158d325: Deleting consensus metadata
I20260926 02:12:00.689924 5381 ts_tablet_manager.cc:1939] T 6351ff0e950245408237a77ab618ffed P 9f131f83858c48aba71713b6e1a854ee: Deleting consensus metadata
I20260926 02:12:00.690007 5380 tablet_replica.cc:330] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47: stopping tablet replica
I20260926 02:12:00.690578 5380 raft_consensus.cc:2229] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:12:00.691237 5380 raft_consensus.cc:2258] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:12:00.691942 4859 catalog_manager.cc:4841] TS 9f131f83858c48aba71713b6e1a854ee (127.18.159.129:39611): tablet 6351ff0e950245408237a77ab618ffed (table test-table [id=5294400afd6747338aff0c4965900151]) successfully deleted
I20260926 02:12:00.691885 4862 catalog_manager.cc:4841] TS 1890c7de3b05421b94c9ccad9158d325 (127.18.159.130:40149): tablet 6351ff0e950245408237a77ab618ffed (table test-table [id=5294400afd6747338aff0c4965900151]) successfully deleted
I20260926 02:12:00.693979 5380 ts_tablet_manager.cc:1905] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:12:00.700243 5380 ts_tablet_manager.cc:1918] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:12:00.700433 5380 log.cc:1196] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/6351ff0e950245408237a77ab618ffed
I20260926 02:12:00.701020 5380 ts_tablet_manager.cc:1939] T 6351ff0e950245408237a77ab618ffed P b44acfa7d5254590b4fdf34f15195d47: Deleting consensus metadata
I20260926 02:12:00.701802 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:00.657079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38854 (local address 127.18.159.129:39611)
0926 02:12:00.657356 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:00.657379 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:00.669159 (+ 11780us) server_negotiation.cc:407] Connection header received
0926 02:12:00.669784 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:00.669794 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:00.669927 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:00.670024 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:00.672903 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.674532 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.679303 (+ 4771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.679765 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.679865 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:00.693670 (+ 13805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:00.693687 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:00.693706 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:00.693754 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:00.697940 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:00.699029 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:00.699040 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:00.699044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:00.699213 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:00.699712 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:00.699717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:00.699721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:00.700263 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:00.700590 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:00.701118 (+ 528us) server_negotiation.cc:299] Negotiation successful
0926 02:12:00.701372 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:12:00.702589 5059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:00.703637 4860 catalog_manager.cc:4841] TS b44acfa7d5254590b4fdf34f15195d47 (127.18.159.131:35651): tablet 6351ff0e950245408237a77ab618ffed (table test-table [id=5294400afd6747338aff0c4965900151]) successfully deleted
I20260926 02:12:00.806849 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 4936
I20260926 02:12:00.825397 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 5070
Sep 26 02:12:00 dist-test-slave-03rn krb5kdc[4830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.130@KRBTEST.COM
I20260926 02:12:00.844143 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:00.826324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42438 (local address 127.18.159.130:40149)
0926 02:12:00.826499 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:00.826504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:00.826521 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:12:00.826678 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:00.826682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:00.826744 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:00.826831 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:00.828626 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.829434 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.831383 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.831922 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.832120 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:00.837021 (+ 4901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:00.837041 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:00.837061 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:00.837113 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:00.840954 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:00.841843 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:00.841852 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:00.841854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:00.842013 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:00.842453 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:00.842458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:00.842471 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:00.842863 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:00.843219 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:00.843540 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 02:12:00.843799 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:12:00.844938 5194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:00.971613 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 5070
I20260926 02:12:00.991477 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 5205
Sep 26 02:12:00 dist-test-slave-03rn krb5kdc[4830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.131@KRBTEST.COM
I20260926 02:12:01.007860 5350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:00.992436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37368 (local address 127.18.159.131:35651)
0926 02:12:00.992604 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:00.992612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:00.992653 (+ 41us) server_negotiation.cc:407] Connection header received
0926 02:12:00.992920 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:00.992926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:00.993026 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:00.993145 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:00.994481 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.995178 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.996547 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:00.996974 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:00.997119 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:01.001925 (+ 4806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:01.001943 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:01.001959 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:01.002000 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:01.004957 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:01.005741 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:01.005750 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:01.005753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:01.005885 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:01.006312 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:01.006316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:01.006318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:01.006673 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:01.006988 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:01.007304 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 02:12:01.007547 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 02:12:01.008582 5329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:01.098121 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 5205
I20260926 02:12:01.114563 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 4843
I20260926 02:12:01.115720 4905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:01.228700 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 4843
2026-09-26T02:12:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4525 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T02:12:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:01Z Disabled control of system clock
I20260926 02:12:01.308269 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:33977
--webserver_interface=127.18.159.190
--webserver_port=0
--builtin_ntp_servers=127.18.159.148:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:33977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:12:01.662462 5391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:01.662813 5391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:01.662978 5391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:01.670532 5391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:01.670663 5391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:01.670746 5391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:01.670835 5391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:01.670905 5391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:01.680714 5391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:33977
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:33977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:01.682123 5391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:01.684108 5391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:01.697240 5396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:01.697157 5397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:01.697196 5399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:01.698438 5391 server_base.cc:1034] running on GCE node
I20260926 02:12:01.699366 5391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:01.701028 5391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:01.702272 5391 hybrid_clock.cc:648] HybridClock initialized: now 1790388721702228 us; error 74 us; skew 500 ppm
I20260926 02:12:01.702658 5391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:12:01.705094 5391 webserver.cc:460] Webserver started at http://127.18.159.190:33691/ using document root <none> and password file <none>
I20260926 02:12:01.705825 5391 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:01.705973 5391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:01.706310 5391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:01.709503 5391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "2b4a5468c7e5452a8aabb44108130675"
format_stamp: "Formatted at 2026-09-26 02:12:01 on dist-test-slave-03rn"
server_key: "046bc99604eb8fcbee7b22bc711061bc"
server_key_iv: "c6ffc9d7d1fc7877329ed811b239b554"
server_key_version: "encryptionkey@0"
I20260926 02:12:01.710374 5391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "2b4a5468c7e5452a8aabb44108130675"
format_stamp: "Formatted at 2026-09-26 02:12:01 on dist-test-slave-03rn"
server_key: "046bc99604eb8fcbee7b22bc711061bc"
server_key_iv: "c6ffc9d7d1fc7877329ed811b239b554"
server_key_version: "encryptionkey@0"
I20260926 02:12:01.715901 5391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:12:01.719769 5405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:01.721305 5391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 02:12:01.721519 5391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "2b4a5468c7e5452a8aabb44108130675"
format_stamp: "Formatted at 2026-09-26 02:12:01 on dist-test-slave-03rn"
server_key: "046bc99604eb8fcbee7b22bc711061bc"
server_key_iv: "c6ffc9d7d1fc7877329ed811b239b554"
server_key_version: "encryptionkey@0"
I20260926 02:12:01.721753 5391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:01.753970 5391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:01.754895 5391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:01.755209 5391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:01.775246 5391 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:33977
I20260926 02:12:01.775270 5456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:33977 every 8 connection(s)
I20260926 02:12:01.776718 5391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:01.780565 5457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:01.785060 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 5391
I20260926 02:12:01.785238 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:01.785773 19070 external_mini_cluster.cc:1428] Setting key 2e41e3bc2ec1a5e1c45108965b3a4b96
I20260926 02:12:01.792588 5457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675: Bootstrap starting.
I20260926 02:12:01.795848 5457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:01.797029 5457 log.cc:824] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:01.800024 5457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675: No bootstrap required, opened a new log
I20260926 02:12:01.806636 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:01.787857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59350 (local address 127.18.159.190:33977)
0926 02:12:01.788949 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:01.788961 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:01.789005 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:12:01.791163 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:01.791191 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:01.791712 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:01.792151 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:01.792767 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:01.794098 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:01.795514 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:01.796391 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:01.796591 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:01.803990 (+ 7399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:01.803999 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:01.804016 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:01.804693 (+ 677us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:01.804789 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:01.804819 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:01.804991 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:01.805236 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 02:12:01.805599 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":402,"threads_started":1}
I20260926 02:12:01.808408 5457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4a5468c7e5452a8aabb44108130675" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 33977 } }
I20260926 02:12:01.808810 5457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:01.808943 5457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b4a5468c7e5452a8aabb44108130675, State: Initialized, Role: FOLLOWER
I20260926 02:12:01.809599 5457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4a5468c7e5452a8aabb44108130675" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 33977 } }
I20260926 02:12:01.809926 5457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:12:01.810103 5457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:12:01.810292 5457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:01.812785 5457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4a5468c7e5452a8aabb44108130675" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 33977 } }
I20260926 02:12:01.813364 5457 leader_election.cc:302] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b4a5468c7e5452a8aabb44108130675; no voters:
I20260926 02:12:01.813997 5457 leader_election.cc:288] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:12:01.814167 5462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:01.816191 5462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675 [term 1 LEADER]: Becoming Leader. State: Replica: 2b4a5468c7e5452a8aabb44108130675, State: Running, Role: LEADER
I20260926 02:12:01.816669 5457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:01.816946 5462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4a5468c7e5452a8aabb44108130675" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 33977 } }
I20260926 02:12:01.819973 5464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b4a5468c7e5452a8aabb44108130675. Latest consensus state: current_term: 1 leader_uuid: "2b4a5468c7e5452a8aabb44108130675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4a5468c7e5452a8aabb44108130675" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 33977 } } }
I20260926 02:12:01.819842 5463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b4a5468c7e5452a8aabb44108130675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4a5468c7e5452a8aabb44108130675" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 33977 } } }
I20260926 02:12:01.820365 5464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:01.820367 5463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:01.823081 5469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:01.828069 5469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:01.837103 5469 catalog_manager.cc:1347] Generated new cluster ID: 304db07607294f5fae8e17daa3a99434
I20260926 02:12:01.837281 5469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:01.852015 5469 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:01.854024 5469 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:01.865403 5469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675: Generated new TSK 0
I20260926 02:12:01.866539 5469 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:12:01.879760 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:33977
--builtin_ntp_servers=127.18.159.148:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:12:02.227397 5481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:02.227691 5481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:02.227782 5481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:02.235356 5481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:02.235474 5481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:02.235587 5481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:12:02.245338 5481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:33977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:02.246770 5481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:02.248737 5481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:02.262523 5486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:02.262562 5489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:02.262584 5487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:02.264043 5481 server_base.cc:1034] running on GCE node
I20260926 02:12:02.264818 5481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:02.266008 5481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:02.267266 5481 hybrid_clock.cc:648] HybridClock initialized: now 1790388722267201 us; error 62 us; skew 500 ppm
I20260926 02:12:02.267632 5481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:12:02.270332 5481 webserver.cc:460] Webserver started at http://127.18.159.129:39275/ using document root <none> and password file <none>
I20260926 02:12:02.271176 5481 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:02.271318 5481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:02.271653 5481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:02.274950 5481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "53b84c62c26d4dcd8f5b9ee55b3221b0"
format_stamp: "Formatted at 2026-09-26 02:12:02 on dist-test-slave-03rn"
server_key: "9e6c76119ffcae525fab923dbd3ac23f"
server_key_iv: "62094ecfccc1967514e74e76ddbeda55"
server_key_version: "encryptionkey@0"
I20260926 02:12:02.275823 5481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "53b84c62c26d4dcd8f5b9ee55b3221b0"
format_stamp: "Formatted at 2026-09-26 02:12:02 on dist-test-slave-03rn"
server_key: "9e6c76119ffcae525fab923dbd3ac23f"
server_key_iv: "62094ecfccc1967514e74e76ddbeda55"
server_key_version: "encryptionkey@0"
I20260926 02:12:02.281563 5481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:12:02.285508 5495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:02.287112 5481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260926 02:12:02.287361 5481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "53b84c62c26d4dcd8f5b9ee55b3221b0"
format_stamp: "Formatted at 2026-09-26 02:12:02 on dist-test-slave-03rn"
server_key: "9e6c76119ffcae525fab923dbd3ac23f"
server_key_iv: "62094ecfccc1967514e74e76ddbeda55"
server_key_version: "encryptionkey@0"
I20260926 02:12:02.287606 5481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:02.302830 5481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:02.303720 5481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:02.304037 5481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:02.305058 5481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:12:02.306620 5481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:12:02.306710 5481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:02.306876 5481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:12:02.306977 5481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:02.341328 5481 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:40599
I20260926 02:12:02.341426 5608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:40599 every 8 connection(s)
I20260926 02:12:02.342721 5481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:12:02.345353 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 5481
I20260926 02:12:02.345546 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:12:02.346166 19070 external_mini_cluster.cc:1428] Setting key b4465c3bb5d684787581b8179710e815
I20260926 02:12:02.349450 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:33977
--builtin_ntp_servers=127.18.159.148:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:12:02.358551 5611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:02.345513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:51645 (local address 127.18.159.190:33977)
0926 02:12:02.346400 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:02.346406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:02.347557 (+ 1151us) server_negotiation.cc:407] Connection header received
0926 02:12:02.350169 (+ 2612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:02.350174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:02.350242 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:02.350339 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:02.351808 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:02.352653 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:02.355163 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:02.355553 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:02.355652 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:02.357434 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:02.357440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:02.357444 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:02.357471 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:02.357517 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:02.357530 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:02.357601 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:02.358074 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 02:12:02.358177 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":582,"threads_started":1}
I20260926 02:12:02.359737 5612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:02.346359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:33977 (local address 127.18.159.129:51645)
0926 02:12:02.347408 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:02.347451 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:02.349853 (+ 2402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:02.350641 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:02.350655 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:02.350791 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:02.351622 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:02.351651 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:02.352809 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:02.352824 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:02.355010 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:02.355028 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:02.355792 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:02.355801 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:02.356189 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:02.357007 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:12:02.357013 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:02.357088 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:12:02.357205 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:02.357638 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:02.357652 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:02.357664 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:12:02.357932 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 02:12:02.358466 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":307,"threads_started":1}
I20260926 02:12:02.361567 5609 heartbeater.cc:347] Connected to a master server at 127.18.159.190:33977
I20260926 02:12:02.361917 5609 heartbeater.cc:464] Registering TS with master...
I20260926 02:12:02.362923 5609 heartbeater.cc:511] Master 127.18.159.190:33977 requested a full tablet report, sending...
I20260926 02:12:02.365288 5422 ts_manager.cc:194] Registered new tserver with Master: 53b84c62c26d4dcd8f5b9ee55b3221b0 (127.18.159.129:40599)
I20260926 02:12:02.367729 5422 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.159.129:51645
W20260926 02:12:02.695942 5613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:02.696394 5613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:02.696558 5613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:02.704125 5613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:02.704347 5613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:02.704535 5613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:12:02.714184 5613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:33977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:02.715566 5613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:02.717607 5613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:02.732223 5621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:02.732273 5619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:02.732455 5618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:02.733342 5613 server_base.cc:1034] running on GCE node
I20260926 02:12:02.734067 5613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:02.735095 5613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:02.736341 5613 hybrid_clock.cc:648] HybridClock initialized: now 1790388722736283 us; error 53 us; skew 500 ppm
I20260926 02:12:02.736680 5613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:12:02.739117 5613 webserver.cc:460] Webserver started at http://127.18.159.130:37259/ using document root <none> and password file <none>
I20260926 02:12:02.739866 5613 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:02.740002 5613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:02.740389 5613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:02.743361 5613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "b7ab10c200ca47d5a9838e206a6f62ce"
format_stamp: "Formatted at 2026-09-26 02:12:02 on dist-test-slave-03rn"
server_key: "291f046a659e7a733ad57f72c3f2591a"
server_key_iv: "a13ac4e452041a1c8900c4c1eba84adc"
server_key_version: "encryptionkey@0"
I20260926 02:12:02.744282 5613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "b7ab10c200ca47d5a9838e206a6f62ce"
format_stamp: "Formatted at 2026-09-26 02:12:02 on dist-test-slave-03rn"
server_key: "291f046a659e7a733ad57f72c3f2591a"
server_key_iv: "a13ac4e452041a1c8900c4c1eba84adc"
server_key_version: "encryptionkey@0"
I20260926 02:12:02.749658 5613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 02:12:02.753443 5627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:02.754884 5613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:12:02.755088 5613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "b7ab10c200ca47d5a9838e206a6f62ce"
format_stamp: "Formatted at 2026-09-26 02:12:02 on dist-test-slave-03rn"
server_key: "291f046a659e7a733ad57f72c3f2591a"
server_key_iv: "a13ac4e452041a1c8900c4c1eba84adc"
server_key_version: "encryptionkey@0"
I20260926 02:12:02.755316 5613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:02.787818 5613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:02.788759 5613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:02.789093 5613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:02.790132 5613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:12:02.791719 5613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:12:02.791841 5613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:02.791970 5613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:12:02.792055 5613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:02.825650 5613 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:45923
I20260926 02:12:02.825698 5741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:45923 every 8 connection(s)
I20260926 02:12:02.827044 5613 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:12:02.832582 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 5613
I20260926 02:12:02.832774 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:12:02.833324 19070 external_mini_cluster.cc:1428] Setting key 03352e404fb4505910ff5558e9d87330
I20260926 02:12:02.836483 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:33977
--builtin_ntp_servers=127.18.159.148:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:12:02.841549 5611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:02.829530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:45829 (local address 127.18.159.190:33977)
0926 02:12:02.829707 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:02.829711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:02.831135 (+ 1424us) server_negotiation.cc:407] Connection header received
0926 02:12:02.833346 (+ 2211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:02.833350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:02.833459 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:02.833558 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:02.834892 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:02.835799 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:02.838459 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:02.838842 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:02.838941 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:02.840692 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:02.840697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:02.840700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:02.840721 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:02.840760 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:02.840774 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:02.840865 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:02.841184 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 02:12:02.841307 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:12:02.843097 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:02.830105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:33977 (local address 127.18.159.130:45829)
0926 02:12:02.830975 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:02.831016 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:02.833023 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:02.833856 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:02.833867 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:02.834001 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:02.834726 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:02.834743 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:02.835958 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:02.835964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:02.838275 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:02.838294 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:02.839074 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:02.839084 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:02.839443 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:02.840273 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:12:02.840279 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:02.840347 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:12:02.840461 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:02.840894 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:02.840907 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:02.840919 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:12:02.841108 (+ 189us) client_negotiation.cc:240] Negotiation successful
0926 02:12:02.841573 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":274,"threads_started":1}
I20260926 02:12:02.844700 5742 heartbeater.cc:347] Connected to a master server at 127.18.159.190:33977
I20260926 02:12:02.845039 5742 heartbeater.cc:464] Registering TS with master...
I20260926 02:12:02.845887 5742 heartbeater.cc:511] Master 127.18.159.190:33977 requested a full tablet report, sending...
I20260926 02:12:02.847424 5422 ts_manager.cc:194] Registered new tserver with Master: b7ab10c200ca47d5a9838e206a6f62ce (127.18.159.130:45923)
I20260926 02:12:02.848726 5422 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.159.130:45829
W20260926 02:12:03.172566 5745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:03.172854 5745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:03.172937 5745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:03.180176 5745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:03.180297 5745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:03.180403 5745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:12:03.189808 5745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:33977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:03.191151 5745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:03.193145 5745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:03.206784 5753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:03.206776 5751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:03.206769 5750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:03.207182 5745 server_base.cc:1034] running on GCE node
I20260926 02:12:03.208014 5745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:03.209062 5745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:03.210317 5745 hybrid_clock.cc:648] HybridClock initialized: now 1790388723210267 us; error 38 us; skew 500 ppm
I20260926 02:12:03.210739 5745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:12:03.213186 5745 webserver.cc:460] Webserver started at http://127.18.159.131:41087/ using document root <none> and password file <none>
I20260926 02:12:03.213941 5745 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:03.214082 5745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:03.214414 5745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:03.217650 5745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "827e96871a4e49589538d019cb7e4aa5"
format_stamp: "Formatted at 2026-09-26 02:12:03 on dist-test-slave-03rn"
server_key: "c785a694ce82b94d966d06103b39170a"
server_key_iv: "67ba3bb751a727d761ab66be61829db8"
server_key_version: "encryptionkey@0"
I20260926 02:12:03.218531 5745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "827e96871a4e49589538d019cb7e4aa5"
format_stamp: "Formatted at 2026-09-26 02:12:03 on dist-test-slave-03rn"
server_key: "c785a694ce82b94d966d06103b39170a"
server_key_iv: "67ba3bb751a727d761ab66be61829db8"
server_key_version: "encryptionkey@0"
I20260926 02:12:03.224068 5745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260926 02:12:03.228024 5759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:03.229539 5745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:12:03.229729 5745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "827e96871a4e49589538d019cb7e4aa5"
format_stamp: "Formatted at 2026-09-26 02:12:03 on dist-test-slave-03rn"
server_key: "c785a694ce82b94d966d06103b39170a"
server_key_iv: "67ba3bb751a727d761ab66be61829db8"
server_key_version: "encryptionkey@0"
I20260926 02:12:03.229959 5745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:03.262392 5745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:03.263309 5745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:03.263649 5745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:03.264802 5745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:12:03.266381 5745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:12:03.266520 5745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:03.266654 5745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:12:03.266754 5745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:03.301677 5745 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:41887
I20260926 02:12:03.301784 5872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:41887 every 8 connection(s)
I20260926 02:12:03.303043 5745 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:12:03.304800 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 5745
I20260926 02:12:03.304984 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:12:03.305416 19070 external_mini_cluster.cc:1428] Setting key edaf8cbee4a89367bc472c3a11133d20
I20260926 02:12:03.317096 5611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.306017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:56737 (local address 127.18.159.190:33977)
0926 02:12:03.306229 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:03.306238 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:03.308217 (+ 1979us) server_negotiation.cc:407] Connection header received
0926 02:12:03.310148 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:03.310152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:03.310219 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:03.310304 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:03.311375 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.312338 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.314069 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.314490 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.314588 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:03.316111 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:03.316116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:03.316119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:03.316141 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:03.316249 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:03.316268 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:03.316349 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:03.316675 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 02:12:03.316765 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:12:03.318315 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.306822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:33977 (local address 127.18.159.131:56737)
0926 02:12:03.308001 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:03.308047 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:03.309883 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:03.310510 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:03.310518 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:03.310607 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:03.311198 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:03.311214 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.312577 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.312581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:03.313902 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:03.313915 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.314727 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.314733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:03.314994 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:03.315769 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:12:03.315773 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:03.315817 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:12:03.315908 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:03.316414 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:03.316429 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:03.316438 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 02:12:03.316631 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 02:12:03.317001 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":329,"threads_started":1}
I20260926 02:12:03.320031 5873 heartbeater.cc:347] Connected to a master server at 127.18.159.190:33977
I20260926 02:12:03.320470 5873 heartbeater.cc:464] Registering TS with master...
I20260926 02:12:03.321573 5873 heartbeater.cc:511] Master 127.18.159.190:33977 requested a full tablet report, sending...
I20260926 02:12:03.322990 5422 ts_manager.cc:194] Registered new tserver with Master: 827e96871a4e49589538d019cb7e4aa5 (127.18.159.131:41887)
I20260926 02:12:03.323876 5422 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.159.131:56737
I20260926 02:12:03.334235 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:12:03.346068 5611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.338607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59354 (local address 127.18.159.190:33977)
0926 02:12:03.338778 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:03.338782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:03.339164 (+ 382us) server_negotiation.cc:407] Connection header received
0926 02:12:03.339430 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:03.339435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:03.339516 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:03.339608 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:03.340101 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.341008 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.343082 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.343697 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.343889 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:03.345246 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:03.345253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:03.345258 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:03.345282 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:03.345321 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:03.345337 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:03.345482 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:03.345689 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 02:12:03.345749 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:12:03.350306 5422 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:12:03.357998 5422 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:12:03.372042 5609 heartbeater.cc:503] Master 127.18.159.190:33977 was elected leader, sending a full tablet report...
I20260926 02:12:03.383607 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.370895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40599 (local address 127.0.0.1:54148)
0926 02:12:03.373173 (+ 2278us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:03.373225 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:03.373459 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:03.375994 (+ 2535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:03.376004 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:03.376046 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:03.376564 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:03.376579 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.378479 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.378486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:03.380983 (+ 2497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:03.380996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.382598 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.382605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:03.382806 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:03.382866 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 02:12:03.382999 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 02:12:03.383198 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1919,"mutex_wait_us":210,"thread_start_us":5170,"threads_started":1}
I20260926 02:12:03.383934 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.371367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54148 (local address 127.18.159.129:40599)
0926 02:12:03.374608 (+ 3241us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:03.374617 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:03.374653 (+ 36us) server_negotiation.cc:407] Connection header received
0926 02:12:03.375425 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:03.375435 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:03.375725 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:03.376774 (+ 1049us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:03.377037 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.378336 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.381190 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.382385 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.382606 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:03.382861 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:03.383309 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 02:12:03.383520 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3075,"thread_start_us":2952,"threads_started":1}
I20260926 02:12:03.385334 5890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.374282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.18.159.131:41887)
0926 02:12:03.374974 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:03.374982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:03.375926 (+ 944us) server_negotiation.cc:407] Connection header received
0926 02:12:03.376218 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:03.376226 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:03.376450 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:03.376604 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:03.377621 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.379129 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.382028 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.383266 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.383450 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:03.383658 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:03.384801 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0926 02:12:03.384977 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":370,"threads_started":1}
I20260926 02:12:03.385335 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.371583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42490 (local address 127.18.159.130:45923)
0926 02:12:03.374127 (+ 2544us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:03.374139 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:03.375585 (+ 1446us) server_negotiation.cc:407] Connection header received
0926 02:12:03.375935 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:03.375944 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:03.376212 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:03.376377 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:03.377363 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.378938 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.382013 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.383428 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.383578 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:03.383786 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:03.384805 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0926 02:12:03.384992 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2336,"thread_start_us":2857,"threads_started":1}
I20260926 02:12:03.385933 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.373673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:41887 (local address 127.0.0.1:47796)
0926 02:12:03.375824 (+ 2151us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:03.375841 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:03.376016 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:03.376935 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:03.376941 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:03.376967 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:03.377429 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:03.377445 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.379344 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.379350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:03.381771 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:03.381786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.384342 (+ 2556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.384349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:03.384572 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:03.384623 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 02:12:03.385534 (+ 911us) client_negotiation.cc:240] Negotiation successful
0926 02:12:03.385680 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2005,"mutex_wait_us":74,"thread_start_us":1471,"threads_started":1}
I20260926 02:12:03.386178 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.371122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:45923 (local address 127.0.0.1:42490)
0926 02:12:03.375447 (+ 4325us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:03.375473 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:03.375707 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:03.376655 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:03.376661 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:03.376689 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:03.377128 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:03.377146 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.379096 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.379102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:03.381771 (+ 2669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:03.381786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.384288 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.384298 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:03.384519 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:03.384617 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 02:12:03.385651 (+ 1034us) client_negotiation.cc:240] Negotiation successful
0926 02:12:03.385790 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4185,"mutex_wait_us":207,"thread_start_us":699,"threads_started":1}
I20260926 02:12:03.388674 5544 tablet_service.cc:1467] Processing CreateTablet for tablet 8350fe692bce48f69573957f9be6adf1 (DEFAULT_TABLE table=test-table [id=b5e414bd43994bbeadf5689b7c58997b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:12:03.389721 5807 tablet_service.cc:1467] Processing CreateTablet for tablet 8350fe692bce48f69573957f9be6adf1 (DEFAULT_TABLE table=test-table [id=b5e414bd43994bbeadf5689b7c58997b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:12:03.390758 5544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8350fe692bce48f69573957f9be6adf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:03.391112 5807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8350fe692bce48f69573957f9be6adf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:03.390614 5677 tablet_service.cc:1467] Processing CreateTablet for tablet 8350fe692bce48f69573957f9be6adf1 (DEFAULT_TABLE table=test-table [id=b5e414bd43994bbeadf5689b7c58997b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:12:03.391913 5677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8350fe692bce48f69573957f9be6adf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:03.404553 5891 tablet_bootstrap.cc:492] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0: Bootstrap starting.
I20260926 02:12:03.407517 5893 tablet_bootstrap.cc:492] T 8350fe692bce48f69573957f9be6adf1 P b7ab10c200ca47d5a9838e206a6f62ce: Bootstrap starting.
I20260926 02:12:03.408910 5891 tablet_bootstrap.cc:654] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:03.410408 5891 log.cc:824] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:03.411662 5892 tablet_bootstrap.cc:492] T 8350fe692bce48f69573957f9be6adf1 P 827e96871a4e49589538d019cb7e4aa5: Bootstrap starting.
I20260926 02:12:03.411785 5893 tablet_bootstrap.cc:654] T 8350fe692bce48f69573957f9be6adf1 P b7ab10c200ca47d5a9838e206a6f62ce: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:03.413957 5893 log.cc:824] T 8350fe692bce48f69573957f9be6adf1 P b7ab10c200ca47d5a9838e206a6f62ce: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:03.416261 5892 tablet_bootstrap.cc:654] T 8350fe692bce48f69573957f9be6adf1 P 827e96871a4e49589538d019cb7e4aa5: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:03.417594 5893 tablet_bootstrap.cc:492] T 8350fe692bce48f69573957f9be6adf1 P b7ab10c200ca47d5a9838e206a6f62ce: No bootstrap required, opened a new log
I20260926 02:12:03.417763 5892 log.cc:824] T 8350fe692bce48f69573957f9be6adf1 P 827e96871a4e49589538d019cb7e4aa5: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:03.417868 5891 tablet_bootstrap.cc:492] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0: No bootstrap required, opened a new log
I20260926 02:12:03.417953 5893 ts_tablet_manager.cc:1397] T 8350fe692bce48f69573957f9be6adf1 P b7ab10c200ca47d5a9838e206a6f62ce: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.007s
I20260926 02:12:03.418143 5891 ts_tablet_manager.cc:1397] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260926 02:12:03.421471 5892 tablet_bootstrap.cc:492] T 8350fe692bce48f69573957f9be6adf1 P 827e96871a4e49589538d019cb7e4aa5: No bootstrap required, opened a new log
I20260926 02:12:03.421779 5892 ts_tablet_manager.cc:1397] T 8350fe692bce48f69573957f9be6adf1 P 827e96871a4e49589538d019cb7e4aa5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 02:12:03.427181 5893 raft_consensus.cc:348] T 8350fe692bce48f69573957f9be6adf1 P b7ab10c200ca47d5a9838e206a6f62ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b84c62c26d4dcd8f5b9ee55b3221b0" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40599 } } peers { permanent_uuid: "b7ab10c200ca47d5a9838e206a6f62ce" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45923 } } peers { permanent_uuid: "827e96871a4e49589538d019cb7e4aa5" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41887 } }
I20260926 02:12:03.427639 5893 raft_consensus.cc:374] T 8350fe692bce48f69573957f9be6adf1 P b7ab10c200ca47d5a9838e206a6f62ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:03.427740 5893 raft_consensus.cc:729] T 8350fe692bce48f69573957f9be6adf1 P b7ab10c200ca47d5a9838e206a6f62ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7ab10c200ca47d5a9838e206a6f62ce, State: Initialized, Role: FOLLOWER
I20260926 02:12:03.428454 5893 consensus_queue.cc:260] T 8350fe692bce48f69573957f9be6adf1 P b7ab10c200ca47d5a9838e206a6f62ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b84c62c26d4dcd8f5b9ee55b3221b0" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40599 } } peers { permanent_uuid: "b7ab10c200ca47d5a9838e206a6f62ce" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45923 } } peers { permanent_uuid: "827e96871a4e49589538d019cb7e4aa5" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41887 } }
I20260926 02:12:03.429227 5891 raft_consensus.cc:348] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b84c62c26d4dcd8f5b9ee55b3221b0" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40599 } } peers { permanent_uuid: "b7ab10c200ca47d5a9838e206a6f62ce" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45923 } } peers { permanent_uuid: "827e96871a4e49589538d019cb7e4aa5" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41887 } }
I20260926 02:12:03.429598 5742 heartbeater.cc:503] Master 127.18.159.190:33977 was elected leader, sending a full tablet report...
I20260926 02:12:03.429698 5891 raft_consensus.cc:374] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:03.429804 5891 raft_consensus.cc:729] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53b84c62c26d4dcd8f5b9ee55b3221b0, State: Initialized, Role: FOLLOWER
I20260926 02:12:03.430429 5891 consensus_queue.cc:260] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b84c62c26d4dcd8f5b9ee55b3221b0" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40599 } } peers { permanent_uuid: "b7ab10c200ca47d5a9838e206a6f62ce" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45923 } } peers { permanent_uuid: "827e96871a4e49589538d019cb7e4aa5" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41887 } }
I20260926 02:12:03.431900 5893 ts_tablet_manager.cc:1428] T 8350fe692bce48f69573957f9be6adf1 P b7ab10c200ca47d5a9838e206a6f62ce: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 02:12:03.434124 5891 ts_tablet_manager.cc:1428] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0: Time spent starting tablet: real 0.016s user 0.014s sys 0.003s
I20260926 02:12:03.433734 5892 raft_consensus.cc:348] T 8350fe692bce48f69573957f9be6adf1 P 827e96871a4e49589538d019cb7e4aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b84c62c26d4dcd8f5b9ee55b3221b0" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40599 } } peers { permanent_uuid: "b7ab10c200ca47d5a9838e206a6f62ce" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45923 } } peers { permanent_uuid: "827e96871a4e49589538d019cb7e4aa5" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41887 } }
I20260926 02:12:03.434289 5892 raft_consensus.cc:374] T 8350fe692bce48f69573957f9be6adf1 P 827e96871a4e49589538d019cb7e4aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:03.434391 5892 raft_consensus.cc:729] T 8350fe692bce48f69573957f9be6adf1 P 827e96871a4e49589538d019cb7e4aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827e96871a4e49589538d019cb7e4aa5, State: Initialized, Role: FOLLOWER
I20260926 02:12:03.435127 5892 consensus_queue.cc:260] T 8350fe692bce48f69573957f9be6adf1 P 827e96871a4e49589538d019cb7e4aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b84c62c26d4dcd8f5b9ee55b3221b0" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40599 } } peers { permanent_uuid: "b7ab10c200ca47d5a9838e206a6f62ce" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45923 } } peers { permanent_uuid: "827e96871a4e49589538d019cb7e4aa5" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41887 } }
I20260926 02:12:03.436992 5892 ts_tablet_manager.cc:1428] T 8350fe692bce48f69573957f9be6adf1 P 827e96871a4e49589538d019cb7e4aa5: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260926 02:12:03.437279 5873 heartbeater.cc:503] Master 127.18.159.190:33977 was elected leader, sending a full tablet report...
I20260926 02:12:03.465785 5898 raft_consensus.cc:482] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:12:03.466070 5898 raft_consensus.cc:504] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b84c62c26d4dcd8f5b9ee55b3221b0" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40599 } } peers { permanent_uuid: "b7ab10c200ca47d5a9838e206a6f62ce" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45923 } } peers { permanent_uuid: "827e96871a4e49589538d019cb7e4aa5" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41887 } }
I20260926 02:12:03.467870 5898 leader_election.cc:288] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7ab10c200ca47d5a9838e206a6f62ce (127.18.159.130:45923), 827e96871a4e49589538d019cb7e4aa5 (127.18.159.131:41887)
I20260926 02:12:03.475270 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.468373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:59755 (local address 127.18.159.130:45923)
0926 02:12:03.468567 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:03.468572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:03.469199 (+ 627us) server_negotiation.cc:407] Connection header received
0926 02:12:03.469496 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:03.469502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:03.469602 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:03.469724 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:03.470548 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.471253 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.473285 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.474266 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.474437 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:03.474506 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:03.474858 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 02:12:03.474995 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:12:03.475359 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.468301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:45923 (local address 127.18.159.129:59755)
0926 02:12:03.469066 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:03.469087 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:03.469237 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:03.469870 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:03.469875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:03.469902 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:03.470384 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:03.470406 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.471408 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.471414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:03.473081 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:03.473093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.474463 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.474470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:03.474654 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:03.474690 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:12:03.474804 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 02:12:03.474949 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":36,"thread_start_us":458,"threads_started":1}
I20260926 02:12:03.475843 5890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.469448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:51077 (local address 127.18.159.131:41887)
0926 02:12:03.469636 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:03.469644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:03.469982 (+ 338us) server_negotiation.cc:407] Connection header received
0926 02:12:03.470072 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:03.470078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:03.470153 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:03.470254 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:03.470886 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.471656 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.473612 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.474702 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.474864 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:03.474928 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:03.475434 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 02:12:03.475581 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:12:03.476578 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.469255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:41887 (local address 127.18.159.129:51077)
0926 02:12:03.469868 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:03.469882 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:03.469978 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:03.470321 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:03.470324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:03.470346 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:03.470677 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:03.470684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.471847 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.471851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:03.473417 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:03.473430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.475065 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.475072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:03.475227 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:03.475259 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 02:12:03.476031 (+ 772us) client_negotiation.cc:240] Negotiation successful
0926 02:12:03.476222 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":17,"thread_start_us":327,"threads_started":1}
I20260926 02:12:03.476617 5697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8350fe692bce48f69573957f9be6adf1" candidate_uuid: "53b84c62c26d4dcd8f5b9ee55b3221b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7ab10c200ca47d5a9838e206a6f62ce" is_pre_election: true
I20260926 02:12:03.477209 5697 raft_consensus.cc:2454] T 8350fe692bce48f69573957f9be6adf1 P b7ab10c200ca47d5a9838e206a6f62ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53b84c62c26d4dcd8f5b9ee55b3221b0 in term 0.
I20260926 02:12:03.477680 5828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8350fe692bce48f69573957f9be6adf1" candidate_uuid: "53b84c62c26d4dcd8f5b9ee55b3221b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "827e96871a4e49589538d019cb7e4aa5" is_pre_election: true
I20260926 02:12:03.477985 5496 leader_election.cc:302] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53b84c62c26d4dcd8f5b9ee55b3221b0, b7ab10c200ca47d5a9838e206a6f62ce; no voters:
I20260926 02:12:03.478281 5828 raft_consensus.cc:2454] T 8350fe692bce48f69573957f9be6adf1 P 827e96871a4e49589538d019cb7e4aa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53b84c62c26d4dcd8f5b9ee55b3221b0 in term 0.
I20260926 02:12:03.478451 5898 raft_consensus.cc:2781] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:12:03.478627 5898 raft_consensus.cc:482] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:12:03.478758 5898 raft_consensus.cc:3037] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:03.481087 5898 raft_consensus.cc:504] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b84c62c26d4dcd8f5b9ee55b3221b0" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40599 } } peers { permanent_uuid: "b7ab10c200ca47d5a9838e206a6f62ce" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45923 } } peers { permanent_uuid: "827e96871a4e49589538d019cb7e4aa5" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41887 } }
I20260926 02:12:03.481711 5898 leader_election.cc:288] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 [CANDIDATE]: Term 1 election: Requested vote from peers b7ab10c200ca47d5a9838e206a6f62ce (127.18.159.130:45923), 827e96871a4e49589538d019cb7e4aa5 (127.18.159.131:41887)
I20260926 02:12:03.482306 5697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8350fe692bce48f69573957f9be6adf1" candidate_uuid: "53b84c62c26d4dcd8f5b9ee55b3221b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7ab10c200ca47d5a9838e206a6f62ce"
I20260926 02:12:03.482518 5697 raft_consensus.cc:3037] T 8350fe692bce48f69573957f9be6adf1 P b7ab10c200ca47d5a9838e206a6f62ce [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:03.482487 5828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8350fe692bce48f69573957f9be6adf1" candidate_uuid: "53b84c62c26d4dcd8f5b9ee55b3221b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "827e96871a4e49589538d019cb7e4aa5"
I20260926 02:12:03.482708 5828 raft_consensus.cc:3037] T 8350fe692bce48f69573957f9be6adf1 P 827e96871a4e49589538d019cb7e4aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:03.485100 5828 raft_consensus.cc:2454] T 8350fe692bce48f69573957f9be6adf1 P 827e96871a4e49589538d019cb7e4aa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53b84c62c26d4dcd8f5b9ee55b3221b0 in term 1.
I20260926 02:12:03.485234 5697 raft_consensus.cc:2454] T 8350fe692bce48f69573957f9be6adf1 P b7ab10c200ca47d5a9838e206a6f62ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53b84c62c26d4dcd8f5b9ee55b3221b0 in term 1.
I20260926 02:12:03.485613 5496 leader_election.cc:302] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53b84c62c26d4dcd8f5b9ee55b3221b0, 827e96871a4e49589538d019cb7e4aa5; no voters:
I20260926 02:12:03.485931 5898 raft_consensus.cc:2781] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:03.486405 5898 raft_consensus.cc:686] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 [term 1 LEADER]: Becoming Leader. State: Replica: 53b84c62c26d4dcd8f5b9ee55b3221b0, State: Running, Role: LEADER
I20260926 02:12:03.486910 5898 consensus_queue.cc:237] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b84c62c26d4dcd8f5b9ee55b3221b0" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40599 } } peers { permanent_uuid: "b7ab10c200ca47d5a9838e206a6f62ce" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45923 } } peers { permanent_uuid: "827e96871a4e49589538d019cb7e4aa5" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41887 } }
I20260926 02:12:03.492311 5421 catalog_manager.cc:5495] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53b84c62c26d4dcd8f5b9ee55b3221b0 (127.18.159.129). New cstate: current_term: 1 leader_uuid: "53b84c62c26d4dcd8f5b9ee55b3221b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b84c62c26d4dcd8f5b9ee55b3221b0" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 40599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7ab10c200ca47d5a9838e206a6f62ce" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "827e96871a4e49589538d019cb7e4aa5" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41887 } health_report { overall_health: UNKNOWN } } }
I20260926 02:12:03.512542 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.506699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54158 (local address 127.18.159.129:40599)
0926 02:12:03.507244 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:03.507249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:03.507274 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:12:03.507391 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:03.507421 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:03.507527 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:03.507650 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:03.508063 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.508914 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.510331 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.510765 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.510871 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:03.511414 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:03.511929 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:03.512061 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:03.512303 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 02:12:03.512343 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424}
I20260926 02:12:03.521770 5828 raft_consensus.cc:1261] T 8350fe692bce48f69573957f9be6adf1 P 827e96871a4e49589538d019cb7e4aa5 [term 1 FOLLOWER]: Refusing update from remote peer 53b84c62c26d4dcd8f5b9ee55b3221b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:12:03.521770 5697 raft_consensus.cc:1261] T 8350fe692bce48f69573957f9be6adf1 P b7ab10c200ca47d5a9838e206a6f62ce [term 1 FOLLOWER]: Refusing update from remote peer 53b84c62c26d4dcd8f5b9ee55b3221b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:12:03.522954 5902 consensus_queue.cc:1035] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7ab10c200ca47d5a9838e206a6f62ce" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:12:03.523423 5898 consensus_queue.cc:1035] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "827e96871a4e49589538d019cb7e4aa5" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 41887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:12:03.550397 5421 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:12:03.550717 5421 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:12:03.554787 5421 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b4a5468c7e5452a8aabb44108130675: Sending DeleteTablet for 3 replicas of tablet 8350fe692bce48f69573957f9be6adf1
I20260926 02:12:03.556113 5677 tablet_service.cc:1514] Processing DeleteTablet for tablet 8350fe692bce48f69573957f9be6adf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:12:03 UTC) from {username='slave'} at 127.0.0.1:42490
I20260926 02:12:03.556730 5544 tablet_service.cc:1514] Processing DeleteTablet for tablet 8350fe692bce48f69573957f9be6adf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:12:03 UTC) from {username='slave'} at 127.0.0.1:54148
I20260926 02:12:03.556943 5807 tablet_service.cc:1514] Processing DeleteTablet for tablet 8350fe692bce48f69573957f9be6adf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:12:03 UTC) from {username='slave'} at 127.0.0.1:47796
I20260926 02:12:03.557723 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 5481
I20260926 02:12:03.558049 5915 tablet_replica.cc:330] T 8350fe692bce48f69573957f9be6adf1 P b7ab10c200ca47d5a9838e206a6f62ce: stopping tablet replica
I20260926 02:12:03.558591 5915 raft_consensus.cc:2229] T 8350fe692bce48f69573957f9be6adf1 P b7ab10c200ca47d5a9838e206a6f62ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:12:03.559269 5915 raft_consensus.cc:2258] T 8350fe692bce48f69573957f9be6adf1 P b7ab10c200ca47d5a9838e206a6f62ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:12:03.568352 5917 tablet_replica.cc:330] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0: stopping tablet replica
I20260926 02:12:03.568970 5917 raft_consensus.cc:2229] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:12:03.569578 5915 ts_tablet_manager.cc:1905] T 8350fe692bce48f69573957f9be6adf1 P b7ab10c200ca47d5a9838e206a6f62ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:12:03.569854 5917 raft_consensus.cc:2258] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:12:03.572242 5916 tablet_replica.cc:330] T 8350fe692bce48f69573957f9be6adf1 P 827e96871a4e49589538d019cb7e4aa5: stopping tablet replica
I20260926 02:12:03.572723 5916 raft_consensus.cc:2229] T 8350fe692bce48f69573957f9be6adf1 P 827e96871a4e49589538d019cb7e4aa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:12:03.573254 5916 raft_consensus.cc:2258] T 8350fe692bce48f69573957f9be6adf1 P 827e96871a4e49589538d019cb7e4aa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:12:03.575558 5917 ts_tablet_manager.cc:1905] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:12:03.577912 5915 ts_tablet_manager.cc:1918] T 8350fe692bce48f69573957f9be6adf1 P b7ab10c200ca47d5a9838e206a6f62ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:12:03.578099 5915 log.cc:1196] T 8350fe692bce48f69573957f9be6adf1 P b7ab10c200ca47d5a9838e206a6f62ce: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/8350fe692bce48f69573957f9be6adf1
I20260926 02:12:03.578706 5915 ts_tablet_manager.cc:1939] T 8350fe692bce48f69573957f9be6adf1 P b7ab10c200ca47d5a9838e206a6f62ce: Deleting consensus metadata
I20260926 02:12:03.580845 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.561775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54174 (local address 127.18.159.129:40599)
0926 02:12:03.562665 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:03.562674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:03.571351 (+ 8677us) server_negotiation.cc:407] Connection header received
0926 02:12:03.571888 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:03.571894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:03.572024 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:03.572109 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:03.573042 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.574297 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.576375 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.577046 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.577167 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:03.578706 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:03.578717 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:03.578735 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:03.579629 (+ 894us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:03.579698 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:03.579723 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:03.579828 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:03.580304 (+ 476us) server_negotiation.cc:299] Negotiation successful
0926 02:12:03.580435 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722}
I20260926 02:12:03.580803 5406 catalog_manager.cc:4841] TS b7ab10c200ca47d5a9838e206a6f62ce (127.18.159.130:45923): tablet 8350fe692bce48f69573957f9be6adf1 (table test-table [id=b5e414bd43994bbeadf5689b7c58997b]) successfully deleted
I20260926 02:12:03.582290 5604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:03.582721 5916 ts_tablet_manager.cc:1905] T 8350fe692bce48f69573957f9be6adf1 P 827e96871a4e49589538d019cb7e4aa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:12:03.590855 5916 ts_tablet_manager.cc:1918] T 8350fe692bce48f69573957f9be6adf1 P 827e96871a4e49589538d019cb7e4aa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:12:03.591070 5916 log.cc:1196] T 8350fe692bce48f69573957f9be6adf1 P 827e96871a4e49589538d019cb7e4aa5: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/8350fe692bce48f69573957f9be6adf1
I20260926 02:12:03.591630 5916 ts_tablet_manager.cc:1939] T 8350fe692bce48f69573957f9be6adf1 P 827e96871a4e49589538d019cb7e4aa5: Deleting consensus metadata
I20260926 02:12:03.593757 5406 catalog_manager.cc:4841] TS 827e96871a4e49589538d019cb7e4aa5 (127.18.159.131:41887): tablet 8350fe692bce48f69573957f9be6adf1 (table test-table [id=b5e414bd43994bbeadf5689b7c58997b]) successfully deleted
I20260926 02:12:03.681162 5917 ts_tablet_manager.cc:1918] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:12:03.681478 5917 log.cc:1196] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/8350fe692bce48f69573957f9be6adf1
I20260926 02:12:03.682049 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 5481
I20260926 02:12:03.682225 5917 ts_tablet_manager.cc:1939] T 8350fe692bce48f69573957f9be6adf1 P 53b84c62c26d4dcd8f5b9ee55b3221b0: Deleting consensus metadata
I20260926 02:12:03.700883 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 5613
W20260926 02:12:03.700912 5409 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.159.129:40599 (error 108)
I20260926 02:12:03.702862 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.702378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40599 (local address 127.0.0.1:54180)
0926 02:12:03.702554 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:03.702684 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 02:12:03.703120 5409 catalog_manager.cc:4584] TS 53b84c62c26d4dcd8f5b9ee55b3221b0 (127.18.159.129:40599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8350fe692bce48f69573957f9be6adf1: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40599: connect: Connection refused (error 111)
I20260926 02:12:03.708319 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.701678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42494 (local address 127.18.159.130:45923)
0926 02:12:03.701851 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:03.701856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:03.702102 (+ 246us) server_negotiation.cc:407] Connection header received
0926 02:12:03.702272 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:03.702276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:03.702373 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:03.702496 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:03.702907 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.703732 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.705082 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.705494 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.705599 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:03.706756 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:03.706765 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:03.706779 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:03.707379 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:03.707427 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:03.707452 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:03.707539 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:03.707863 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 02:12:03.707974 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:12:03.709040 5737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:03.757053 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.756543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40599 (local address 127.0.0.1:54186)
0926 02:12:03.756762 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:03.756877 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 02:12:03.758203 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.757807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40599 (local address 127.0.0.1:54188)
0926 02:12:03.757980 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:03.758065 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 02:12:03.799461 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 5613
I20260926 02:12:03.816438 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 5745
I20260926 02:12:03.819419 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.818883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40599 (local address 127.0.0.1:54204)
0926 02:12:03.819080 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:03.819177 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 02:12:03.820431 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.820071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40599 (local address 127.0.0.1:54212)
0926 02:12:03.820237 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:03.820322 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 02:12:03.823787 5890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.817286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47810 (local address 127.18.159.131:41887)
0926 02:12:03.817504 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:03.817511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:03.817531 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:12:03.817670 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:03.817677 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:03.817785 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:03.817917 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:03.818313 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.819168 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.820665 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:03.821084 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:03.821180 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:03.822297 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:03.822306 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:03.822321 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:03.822935 (+ 614us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:03.823065 (+ 130us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:03.823089 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:03.823180 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:03.823320 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 02:12:03.823452 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 02:12:03.824499 5868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:03.901039 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.900555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40599 (local address 127.0.0.1:54220)
0926 02:12:03.900786 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:03.900874 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 02:12:03.902213 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:03.901799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:40599 (local address 127.0.0.1:54228)
0926 02:12:03.902022 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:03.902094 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.129:40599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 02:12:03.913254 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 5745
I20260926 02:12:03.930400 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 5391
I20260926 02:12:03.931586 5452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:04.042770 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 5391
2026-09-26T02:12:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2812 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:12:04 dist-test-slave-03rn krb5kdc[5926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:12:04 dist-test-slave-03rn krb5kdc[5926](info): set up 2 sockets
Sep 26 02:12:04 dist-test-slave-03rn krb5kdc[5926](info): commencing operation
krb5kdc: starting...
W20260926 02:12:05.184811 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:12:05 dist-test-slave-03rn krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:12:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:12:05.444633 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:39369
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:42079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:39369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:05.794862 5939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:05.795181 5939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:05.795346 5939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:05.803131 5939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:05.803257 5939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:05.803337 5939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:05.803424 5939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:05.803503 5939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:05.813872 5939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:42079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:39369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:39369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:05.815318 5939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:05.816771 5939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:05.829183 5945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:05.829232 5947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:05.829701 5944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:05.830308 5939 server_base.cc:1034] running on GCE node
I20260926 02:12:05.831230 5939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:05.832844 5939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:05.834055 5939 hybrid_clock.cc:648] HybridClock initialized: now 1790388725834014 us; error 27 us; skew 500 ppm
I20260926 02:12:05.834549 5939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:05 dist-test-slave-03rn krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388725, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:05.837833 5939 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:12:05.839578 5939 webserver.cc:460] Webserver started at http://127.18.159.190:46063/ using document root <none> and password file <none>
I20260926 02:12:05.840395 5939 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:05.840549 5939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:05.840873 5939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:05.844027 5939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "050625ce34e4496f88cbf254a6258a14"
format_stamp: "Formatted at 2026-09-26 02:12:05 on dist-test-slave-03rn"
server_key: "a36a8c2f93e942156d8efe5e32787cc4"
server_key_iv: "57c62848b4a4ac5a2e4fdc1e920b86cb"
server_key_version: "encryptionkey@0"
I20260926 02:12:05.844918 5939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "050625ce34e4496f88cbf254a6258a14"
format_stamp: "Formatted at 2026-09-26 02:12:05 on dist-test-slave-03rn"
server_key: "a36a8c2f93e942156d8efe5e32787cc4"
server_key_iv: "57c62848b4a4ac5a2e4fdc1e920b86cb"
server_key_version: "encryptionkey@0"
I20260926 02:12:05.850639 5939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:12:05.854504 5954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:05.856014 5939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:12:05.856279 5939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "050625ce34e4496f88cbf254a6258a14"
format_stamp: "Formatted at 2026-09-26 02:12:05 on dist-test-slave-03rn"
server_key: "a36a8c2f93e942156d8efe5e32787cc4"
server_key_iv: "57c62848b4a4ac5a2e4fdc1e920b86cb"
server_key_version: "encryptionkey@0"
I20260926 02:12:05.856518 5939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:05.867453 5939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:05.871810 5939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:05.872179 5939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:05.892409 5939 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:39369
I20260926 02:12:05.892454 6006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:39369 every 8 connection(s)
I20260926 02:12:05.893895 5939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:05.898043 6007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:05.900718 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 5939
I20260926 02:12:05.900911 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:05.901443 19070 external_mini_cluster.cc:1428] Setting key 8940a605b9c3683f47a4d474185256ee
I20260926 02:12:05.911291 6007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14: Bootstrap starting.
I20260926 02:12:05.915473 6007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14: Neither blocks nor log segments found. Creating new log.
Sep 26 02:12:05 dist-test-slave-03rn krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:12:05.916815 6007 log.cc:824] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:05.920290 6007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14: No bootstrap required, opened a new log
I20260926 02:12:05.925585 6010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:05.903408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52354 (local address 127.18.159.190:39369)
0926 02:12:05.904340 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:05.904353 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:05.904395 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:12:05.906047 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:05.906069 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:05.906482 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:05.906808 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:05.908762 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:05.910102 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:05.911954 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:05.912711 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:05.912859 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:05.917833 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:05.917841 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:05.917857 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:05.917910 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:05.921195 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:05.921972 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:05.921980 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:05.921990 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:05.922184 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:05.922617 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:05.922629 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:05.922632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:05.923216 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:05.923569 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:05.924016 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 02:12:05.924514 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":441,"threads_started":1}
I20260926 02:12:05.929181 6007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050625ce34e4496f88cbf254a6258a14" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39369 } }
I20260926 02:12:05.929600 6007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:05.929711 6007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 050625ce34e4496f88cbf254a6258a14, State: Initialized, Role: FOLLOWER
I20260926 02:12:05.930343 6007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050625ce34e4496f88cbf254a6258a14" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39369 } }
I20260926 02:12:05.930598 6007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:12:05.930737 6007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:12:05.930915 6007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:05.933451 6007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050625ce34e4496f88cbf254a6258a14" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39369 } }
I20260926 02:12:05.933970 6007 leader_election.cc:302] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 050625ce34e4496f88cbf254a6258a14; no voters:
I20260926 02:12:05.934587 6007 leader_election.cc:288] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:12:05.934731 6012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:05.935451 6012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14 [term 1 LEADER]: Becoming Leader. State: Replica: 050625ce34e4496f88cbf254a6258a14, State: Running, Role: LEADER
I20260926 02:12:05.936017 6012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050625ce34e4496f88cbf254a6258a14" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39369 } }
I20260926 02:12:05.936817 6007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:05.938833 6013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "050625ce34e4496f88cbf254a6258a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050625ce34e4496f88cbf254a6258a14" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39369 } } }
I20260926 02:12:05.939276 6013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:05.940269 6014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 050625ce34e4496f88cbf254a6258a14. Latest consensus state: current_term: 1 leader_uuid: "050625ce34e4496f88cbf254a6258a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050625ce34e4496f88cbf254a6258a14" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39369 } } }
I20260926 02:12:05.940795 6014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:05.944971 6017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:05.950436 6017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:05.961344 6017 catalog_manager.cc:1347] Generated new cluster ID: 1733e1317a3f49a881d217fa46bb3b0e
I20260926 02:12:05.961490 6017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:05.993105 6017 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:05.994666 6017 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:06.011618 6017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14: Generated new TSK 0
I20260926 02:12:06.012786 6017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.159.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
Entry for principal kudu/127.18.159.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab.
I20260926 02:12:06.096256 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:39369
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:42079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:06.474422 6033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:06.474733 6033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:06.474915 6033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:06.482509 6033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:06.482642 6033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:06.482833 6033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:12:06.493039 6033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:42079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.129.keytab
--principal=kudu/127.18.159.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:39369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:06.494354 6033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:06.495622 6033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:06.510077 6039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:06.510363 6041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:06.510123 6038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:06.512593 6033 server_base.cc:1034] running on GCE node
I20260926 02:12:06.513563 6033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:06.515005 6033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:06.516265 6033 hybrid_clock.cc:648] HybridClock initialized: now 1790388726516211 us; error 41 us; skew 500 ppm
I20260926 02:12:06.516656 6033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:06 dist-test-slave-03rn krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388726, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:06.520919 6033 init.cc:377] Logged in from keytab as kudu/127.18.159.129@KRBTEST.COM (short username kudu)
I20260926 02:12:06.522475 6033 webserver.cc:460] Webserver started at http://127.18.159.129:33779/ using document root <none> and password file <none>
I20260926 02:12:06.523205 6033 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:06.523316 6033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:06.523589 6033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:06.526794 6033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "ef684965d8d040cdbb44d4e60d1e8df5"
format_stamp: "Formatted at 2026-09-26 02:12:06 on dist-test-slave-03rn"
server_key: "779204cd06e74b9248347d0e8c9aa231"
server_key_iv: "6ab9dfd53c89c026c077d6c787c38369"
server_key_version: "encryptionkey@0"
I20260926 02:12:06.527555 6033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "ef684965d8d040cdbb44d4e60d1e8df5"
format_stamp: "Formatted at 2026-09-26 02:12:06 on dist-test-slave-03rn"
server_key: "779204cd06e74b9248347d0e8c9aa231"
server_key_iv: "6ab9dfd53c89c026c077d6c787c38369"
server_key_version: "encryptionkey@0"
I20260926 02:12:06.533046 6033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 02:12:06.536653 6048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:06.538256 6033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 02:12:06.538554 6033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "ef684965d8d040cdbb44d4e60d1e8df5"
format_stamp: "Formatted at 2026-09-26 02:12:06 on dist-test-slave-03rn"
server_key: "779204cd06e74b9248347d0e8c9aa231"
server_key_iv: "6ab9dfd53c89c026c077d6c787c38369"
server_key_version: "encryptionkey@0"
I20260926 02:12:06.538812 6033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:06.551748 6033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:06.555855 6033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:06.556216 6033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:06.557228 6033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:12:06.558598 6033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:12:06.558706 6033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:06.559022 6033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:12:06.559121 6033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:06.593561 6033 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:42801
I20260926 02:12:06.593605 6161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:42801 every 8 connection(s)
I20260926 02:12:06.594905 6033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:12:06.598397 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 6033
I20260926 02:12:06.598600 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:12:06.599200 19070 external_mini_cluster.cc:1428] Setting key 5db82ee72ccd61b8621e5724a6b0881b
Sep 26 02:12:06 dist-test-slave-03rn krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388726, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.129@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:12:06.620455 6164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:06.597588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:54681 (local address 127.18.159.190:39369)
0926 02:12:06.598179 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:06.598186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:06.599562 (+ 1376us) server_negotiation.cc:407] Connection header received
0926 02:12:06.601969 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:06.601973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:06.602046 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:06.602145 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:06.604434 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:06.605215 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:06.607859 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:06.608383 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:06.608487 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:06.613090 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:06.613099 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:06.613108 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:06.613159 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:06.616805 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:06.617652 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:06.617661 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:06.617668 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:06.617834 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:06.618289 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:06.618297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:06.618303 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:06.618700 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:06.618863 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:06.619637 (+ 774us) server_negotiation.cc:299] Negotiation successful
0926 02:12:06.619927 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":260,"threads_started":1}
I20260926 02:12:06.621167 6165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:06.598364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:39369 (local address 127.18.159.129:54681)
0926 02:12:06.599357 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:06.599409 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:06.601634 (+ 2225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:06.602429 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:06.602441 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:06.603318 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:06.604219 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:06.604249 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:06.605393 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:06.605402 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:06.607657 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:06.607682 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:06.608624 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:06.608633 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:06.609001 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:06.609788 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:06.609824 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:06.612810 (+ 2986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:06.617003 (+ 4193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:06.617011 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:06.617028 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:06.617500 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:06.617941 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:06.617944 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:06.617947 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:06.618154 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:06.618820 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:06.618825 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:06.619180 (+ 355us) client_negotiation.cc:769] Sending connection context
0926 02:12:06.619458 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 02:12:06.619894 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":410,"threads_started":1}
I20260926 02:12:06.622928 6162 heartbeater.cc:347] Connected to a master server at 127.18.159.190:39369
I20260926 02:12:06.623296 6162 heartbeater.cc:464] Registering TS with master...
I20260926 02:12:06.624217 6162 heartbeater.cc:511] Master 127.18.159.190:39369 requested a full tablet report, sending...
I20260926 02:12:06.626642 5972 ts_manager.cc:194] Registered new tserver with Master: ef684965d8d040cdbb44d4e60d1e8df5 (127.18.159.129:42801)
WARNING: no policy specified for kudu/127.18.159.130@KRBTEST.COM; defaulting to no policy
I20260926 02:12:06.628297 5972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.129@KRBTEST.COM'} at 127.18.159.129:54681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
Entry for principal kudu/127.18.159.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab.
I20260926 02:12:06.660387 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:39369
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:42079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:06.996999 6168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:06.997290 6168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:06.997489 6168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:07.005246 6168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:07.005373 6168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:07.005586 6168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:12:07.015519 6168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:42079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.130.keytab
--principal=kudu/127.18.159.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:39369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:07.017001 6168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:07.018322 6168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:07.031490 6173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:07.031558 6174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:07.031520 6176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:07.032464 6168 server_base.cc:1034] running on GCE node
I20260926 02:12:07.033087 6168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:07.034091 6168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:07.035323 6168 hybrid_clock.cc:648] HybridClock initialized: now 1790388727035271 us; error 46 us; skew 500 ppm
I20260926 02:12:07.035645 6168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:07 dist-test-slave-03rn krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388727, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:07.042964 6168 init.cc:377] Logged in from keytab as kudu/127.18.159.130@KRBTEST.COM (short username kudu)
I20260926 02:12:07.044755 6168 webserver.cc:460] Webserver started at http://127.18.159.130:44111/ using document root <none> and password file <none>
I20260926 02:12:07.045490 6168 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:07.045619 6168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:07.045954 6168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:07.048997 6168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "35f93f25fb614a4c870c6b04c3d90f26"
format_stamp: "Formatted at 2026-09-26 02:12:07 on dist-test-slave-03rn"
server_key: "45adbb7b787606f3f7f767d6018c91ef"
server_key_iv: "5c0fa312cdb7014b5951a153e00aec80"
server_key_version: "encryptionkey@0"
I20260926 02:12:07.049865 6168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "35f93f25fb614a4c870c6b04c3d90f26"
format_stamp: "Formatted at 2026-09-26 02:12:07 on dist-test-slave-03rn"
server_key: "45adbb7b787606f3f7f767d6018c91ef"
server_key_iv: "5c0fa312cdb7014b5951a153e00aec80"
server_key_version: "encryptionkey@0"
I20260926 02:12:07.055402 6168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:12:07.059100 6183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:07.060642 6168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:12:07.060819 6168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "35f93f25fb614a4c870c6b04c3d90f26"
format_stamp: "Formatted at 2026-09-26 02:12:07 on dist-test-slave-03rn"
server_key: "45adbb7b787606f3f7f767d6018c91ef"
server_key_iv: "5c0fa312cdb7014b5951a153e00aec80"
server_key_version: "encryptionkey@0"
I20260926 02:12:07.061055 6168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:07.074448 6168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:07.078781 6168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:07.079098 6168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:07.080107 6168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:12:07.081521 6168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:12:07.081591 6168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:07.081753 6168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:12:07.081820 6168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:07.116425 6168 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:35437
I20260926 02:12:07.116483 6295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:35437 every 8 connection(s)
I20260926 02:12:07.117765 6168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:12:07.126271 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 6168
I20260926 02:12:07.126461 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:12:07.127009 19070 external_mini_cluster.cc:1428] Setting key 6f879151525c2cd9dddd4dfc2ba6bbc5
Sep 26 02:12:07 dist-test-slave-03rn krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388727, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.130@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:12:07.141170 6164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:07.120542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:42869 (local address 127.18.159.190:39369)
0926 02:12:07.120746 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:07.120751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:07.122084 (+ 1333us) server_negotiation.cc:407] Connection header received
0926 02:12:07.124096 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:07.124102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:07.124248 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:07.124381 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:07.126212 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.127126 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.129127 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.129784 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.129930 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:07.133573 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:07.133582 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:07.133590 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:07.133641 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:07.137491 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:07.138582 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:07.138587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:07.138590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:07.138732 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:07.139250 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:07.139261 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:07.139263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:07.139565 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:07.139729 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:07.140553 (+ 824us) server_negotiation.cc:299] Negotiation successful
0926 02:12:07.140816 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:12:07.142381 6298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:07.121044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:39369 (local address 127.18.159.130:42869)
0926 02:12:07.121887 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:07.121945 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:07.123770 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:07.124640 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:07.124649 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:07.125235 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:07.125953 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:07.125984 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.127326 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.127330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:07.128933 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:07.128951 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.130034 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.130039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:07.130259 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:07.131002 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:07.131025 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:07.133305 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:07.137717 (+ 4412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:07.137730 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:07.137757 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:07.138419 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:07.138888 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:07.138893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:07.138896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:07.139101 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:07.139700 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:07.139710 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:07.140090 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 02:12:07.140449 (+ 359us) client_negotiation.cc:240] Negotiation successful
0926 02:12:07.141052 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":286,"threads_started":1}
I20260926 02:12:07.143996 6296 heartbeater.cc:347] Connected to a master server at 127.18.159.190:39369
I20260926 02:12:07.144393 6296 heartbeater.cc:464] Registering TS with master...
I20260926 02:12:07.145285 6296 heartbeater.cc:511] Master 127.18.159.190:39369 requested a full tablet report, sending...
I20260926 02:12:07.147032 5972 ts_manager.cc:194] Registered new tserver with Master: 35f93f25fb614a4c870c6b04c3d90f26 (127.18.159.130:35437)
I20260926 02:12:07.147918 5972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.130@KRBTEST.COM'} at 127.18.159.130:42869
WARNING: no policy specified for kudu/127.18.159.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
Entry for principal kudu/127.18.159.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab.
I20260926 02:12:07.186770 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:39369
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:42079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:07.529686 6301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:07.530032 6301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:07.530205 6301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:07.537822 6301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:07.537989 6301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:07.538167 6301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:12:07.548506 6301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:42079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.131.keytab
--principal=kudu/127.18.159.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:39369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:07.549899 6301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:07.551231 6301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:07.565107 6307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:07.565146 6309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:07.565069 6306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:07.566018 6301 server_base.cc:1034] running on GCE node
I20260926 02:12:07.566699 6301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:07.567819 6301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:07.569059 6301 hybrid_clock.cc:648] HybridClock initialized: now 1790388727569010 us; error 44 us; skew 500 ppm
I20260926 02:12:07.569429 6301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:07 dist-test-slave-03rn krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388727, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:07.572779 6301 init.cc:377] Logged in from keytab as kudu/127.18.159.131@KRBTEST.COM (short username kudu)
I20260926 02:12:07.574396 6301 webserver.cc:460] Webserver started at http://127.18.159.131:34219/ using document root <none> and password file <none>
I20260926 02:12:07.575152 6301 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:07.575310 6301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:07.575651 6301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:07.578763 6301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "384d555300994c9797c68cad5970ebb2"
format_stamp: "Formatted at 2026-09-26 02:12:07 on dist-test-slave-03rn"
server_key: "7ac07960f2a3b4df500bb73456076725"
server_key_iv: "8cdbb24abd12f2657335f2c75369958a"
server_key_version: "encryptionkey@0"
I20260926 02:12:07.579598 6301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "384d555300994c9797c68cad5970ebb2"
format_stamp: "Formatted at 2026-09-26 02:12:07 on dist-test-slave-03rn"
server_key: "7ac07960f2a3b4df500bb73456076725"
server_key_iv: "8cdbb24abd12f2657335f2c75369958a"
server_key_version: "encryptionkey@0"
I20260926 02:12:07.585317 6301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:12:07.589352 6316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:07.590885 6301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:12:07.591115 6301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "384d555300994c9797c68cad5970ebb2"
format_stamp: "Formatted at 2026-09-26 02:12:07 on dist-test-slave-03rn"
server_key: "7ac07960f2a3b4df500bb73456076725"
server_key_iv: "8cdbb24abd12f2657335f2c75369958a"
server_key_version: "encryptionkey@0"
I20260926 02:12:07.591374 6301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:07.616328 6301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:07.621398 6301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:07.621733 6301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:07.622782 6301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:12:07.624199 6301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:12:07.624310 6301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:07.624475 6301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:12:07.624548 6301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:07.631839 6162 heartbeater.cc:503] Master 127.18.159.190:39369 was elected leader, sending a full tablet report...
I20260926 02:12:07.661089 6301 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:34697
I20260926 02:12:07.661130 6428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:34697 every 8 connection(s)
I20260926 02:12:07.662530 6301 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:12:07.663597 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 6301
I20260926 02:12:07.663785 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:12:07.664346 19070 external_mini_cluster.cc:1428] Setting key 50ea534ad8899ef57a219d1e7c2d4d0f
Sep 26 02:12:07 dist-test-slave-03rn krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388727, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.131@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:12:07.684486 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:07.665678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:32979 (local address 127.18.159.190:39369)
0926 02:12:07.666198 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:07.666204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:07.667387 (+ 1183us) server_negotiation.cc:407] Connection header received
0926 02:12:07.669441 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:07.669445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:07.669519 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:07.669619 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:07.671214 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.671976 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.673642 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.674054 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.674150 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:07.677761 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:07.677766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:07.677769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:07.677807 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:07.681144 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:07.682023 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:07.682027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:07.682029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:07.682155 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:07.682609 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:07.682613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:07.682615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:07.682894 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:07.683029 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:07.683907 (+ 878us) server_negotiation.cc:299] Negotiation successful
0926 02:12:07.684106 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":253,"threads_started":1}
I20260926 02:12:07.685626 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:07.666316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:39369 (local address 127.18.159.131:32979)
0926 02:12:07.667216 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:07.667266 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:07.669137 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:07.669849 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:07.669859 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:07.670421 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:07.671037 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:07.671059 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.672113 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.672116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:07.673493 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:07.673506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.674259 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.674264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:07.674481 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:07.675180 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:07.675202 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:07.677505 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:07.681315 (+ 3810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:07.681324 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:07.681340 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:07.681889 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:07.682268 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:07.682272 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:07.682275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:07.682498 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:07.683047 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:07.683055 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:07.683476 (+ 421us) client_negotiation.cc:769] Sending connection context
0926 02:12:07.683770 (+ 294us) client_negotiation.cc:240] Negotiation successful
0926 02:12:07.684327 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":391,"threads_started":1}
I20260926 02:12:07.687111 6429 heartbeater.cc:347] Connected to a master server at 127.18.159.190:39369
I20260926 02:12:07.687463 6429 heartbeater.cc:464] Registering TS with master...
I20260926 02:12:07.688309 6429 heartbeater.cc:511] Master 127.18.159.190:39369 requested a full tablet report, sending...
I20260926 02:12:07.689946 5972 ts_manager.cc:194] Registered new tserver with Master: 384d555300994c9797c68cad5970ebb2 (127.18.159.131:34697)
I20260926 02:12:07.690824 5972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.159.131@KRBTEST.COM'} at 127.18.159.131:32979
I20260926 02:12:07.693332 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:12:07.710677 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:07.697865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52358 (local address 127.18.159.190:39369)
0926 02:12:07.698121 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:07.698127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:07.698394 (+ 267us) server_negotiation.cc:407] Connection header received
0926 02:12:07.698540 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:07.698544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:07.698610 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:07.698696 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:07.700182 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.700970 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.702297 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.702783 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.702903 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:07.705441 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:07.705447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:07.705450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:07.705486 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:07.708081 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:07.708838 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:07.708842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:07.708844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:07.708974 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:07.709307 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:07.709310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:07.709312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:07.709663 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:07.709844 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:07.710138 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 02:12:07.710319 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 02:12:07.714802 5972 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:12:07.721742 5972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:12:07.761241 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:07.735001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:34697 (local address 127.0.0.1:59566)
0926 02:12:07.750904 (+ 15903us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:07.750933 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:07.751140 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:07.752962 (+ 1822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:07.752970 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:07.753007 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:07.753490 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:07.753507 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.756033 (+ 2526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.756042 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:07.758170 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:07.758229 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.760101 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.760108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:07.760369 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:07.760456 (+ 87us) client_negotiation.cc:769] Sending connection context
0926 02:12:07.760594 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 02:12:07.760812 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15715,"thread_start_us":11511,"threads_started":1}
I20260926 02:12:07.761353 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:07.735364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59566 (local address 127.18.159.131:34697)
0926 02:12:07.752005 (+ 16641us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:07.752014 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:07.752048 (+ 34us) server_negotiation.cc:407] Connection header received
0926 02:12:07.752149 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:07.752181 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:07.752436 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:07.752622 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:07.753732 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.755825 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.758416 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.759904 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.760106 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:07.760359 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:07.760832 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 02:12:07.761020 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16458,"thread_start_us":16249,"threads_started":1}
I20260926 02:12:07.762442 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:07.734937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:42801 (local address 127.0.0.1:51248)
0926 02:12:07.746941 (+ 12004us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:07.747003 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:07.747216 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:07.752311 (+ 5095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:07.752325 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:07.752376 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:07.752933 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:07.752950 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.755736 (+ 2786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.755743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:07.758711 (+ 2968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:07.758725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.761610 (+ 2885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.761616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:07.761812 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:07.761844 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 02:12:07.761928 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 02:12:07.762075 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11720,"thread_start_us":16310,"threads_started":1}
I20260926 02:12:07.762619 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:07.735148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51248 (local address 127.18.159.129:42801)
0926 02:12:07.750605 (+ 15457us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:07.750620 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:07.750668 (+ 48us) server_negotiation.cc:407] Connection header received
0926 02:12:07.750819 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:07.750833 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:07.751228 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:07.752246 (+ 1018us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:07.753258 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.755527 (+ 2269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.758914 (+ 3387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.760997 (+ 2083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.761192 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:07.761419 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:07.762145 (+ 726us) server_negotiation.cc:299] Negotiation successful
0926 02:12:07.762315 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15185,"thread_start_us":16752,"threads_started":1}
I20260926 02:12:07.764930 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:07.752649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:35437 (local address 127.0.0.1:49158)
0926 02:12:07.756430 (+ 3781us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:07.756451 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:07.756635 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:07.757417 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:07.757423 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:07.757471 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:07.757918 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:07.757929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.760123 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.760128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:07.762410 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:07.762433 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.764340 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.764349 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:07.764536 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:07.764561 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 02:12:07.764616 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 02:12:07.764706 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3637,"thread_start_us":3396,"threads_started":1}
I20260926 02:12:07.765930 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:07.747846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49158 (local address 127.18.159.130:35437)
0926 02:12:07.756002 (+ 8156us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:07.756011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:07.756777 (+ 766us) server_negotiation.cc:407] Connection header received
0926 02:12:07.756887 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:07.756894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:07.757186 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:07.758129 (+ 943us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:07.758358 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.759908 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.762629 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.764071 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.765393 (+ 1322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:07.765522 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:07.765574 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 02:12:07.765693 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7970,"thread_start_us":7806,"threads_started":1}
I20260926 02:12:07.766028 6364 tablet_service.cc:1467] Processing CreateTablet for tablet 892ccbc5c7854e25994d45a7e5bde0f5 (DEFAULT_TABLE table=test-table [id=a8bfc94a89f64adbb0873fcba5cf4ff7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:12:07.766418 6096 tablet_service.cc:1467] Processing CreateTablet for tablet 892ccbc5c7854e25994d45a7e5bde0f5 (DEFAULT_TABLE table=test-table [id=a8bfc94a89f64adbb0873fcba5cf4ff7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:12:07.767915 6096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892ccbc5c7854e25994d45a7e5bde0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:07.768801 6364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892ccbc5c7854e25994d45a7e5bde0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:07.769400 6231 tablet_service.cc:1467] Processing CreateTablet for tablet 892ccbc5c7854e25994d45a7e5bde0f5 (DEFAULT_TABLE table=test-table [id=a8bfc94a89f64adbb0873fcba5cf4ff7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:12:07.770740 6231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892ccbc5c7854e25994d45a7e5bde0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:07.781350 6448 tablet_bootstrap.cc:492] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2: Bootstrap starting.
I20260926 02:12:07.783077 6449 tablet_bootstrap.cc:492] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5: Bootstrap starting.
I20260926 02:12:07.784868 6450 tablet_bootstrap.cc:492] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26: Bootstrap starting.
I20260926 02:12:07.786116 6448 tablet_bootstrap.cc:654] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:07.786708 6449 tablet_bootstrap.cc:654] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:07.788007 6450 tablet_bootstrap.cc:654] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:07.788007 6448 log.cc:824] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:07.788005 6449 log.cc:824] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:07.789181 6450 log.cc:824] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:07.791422 6449 tablet_bootstrap.cc:492] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5: No bootstrap required, opened a new log
I20260926 02:12:07.791421 6448 tablet_bootstrap.cc:492] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2: No bootstrap required, opened a new log
I20260926 02:12:07.791678 6450 tablet_bootstrap.cc:492] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26: No bootstrap required, opened a new log
I20260926 02:12:07.791716 6449 ts_tablet_manager.cc:1397] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 02:12:07.791716 6448 ts_tablet_manager.cc:1397] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260926 02:12:07.791935 6450 ts_tablet_manager.cc:1397] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 02:12:07.799904 6450 raft_consensus.cc:348] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef684965d8d040cdbb44d4e60d1e8df5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42801 } } peers { permanent_uuid: "384d555300994c9797c68cad5970ebb2" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 34697 } } peers { permanent_uuid: "35f93f25fb614a4c870c6b04c3d90f26" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 35437 } }
I20260926 02:12:07.800426 6450 raft_consensus.cc:374] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:07.800532 6450 raft_consensus.cc:729] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f93f25fb614a4c870c6b04c3d90f26, State: Initialized, Role: FOLLOWER
I20260926 02:12:07.801282 6450 consensus_queue.cc:260] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef684965d8d040cdbb44d4e60d1e8df5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42801 } } peers { permanent_uuid: "384d555300994c9797c68cad5970ebb2" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 34697 } } peers { permanent_uuid: "35f93f25fb614a4c870c6b04c3d90f26" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 35437 } }
I20260926 02:12:07.802459 6296 heartbeater.cc:503] Master 127.18.159.190:39369 was elected leader, sending a full tablet report...
I20260926 02:12:07.803218 6450 ts_tablet_manager.cc:1428] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260926 02:12:07.803030 6449 raft_consensus.cc:348] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef684965d8d040cdbb44d4e60d1e8df5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42801 } } peers { permanent_uuid: "384d555300994c9797c68cad5970ebb2" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 34697 } } peers { permanent_uuid: "35f93f25fb614a4c870c6b04c3d90f26" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 35437 } }
I20260926 02:12:07.803053 6448 raft_consensus.cc:348] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef684965d8d040cdbb44d4e60d1e8df5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42801 } } peers { permanent_uuid: "384d555300994c9797c68cad5970ebb2" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 34697 } } peers { permanent_uuid: "35f93f25fb614a4c870c6b04c3d90f26" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 35437 } }
I20260926 02:12:07.803472 6448 raft_consensus.cc:374] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:07.803488 6449 raft_consensus.cc:374] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:07.803588 6448 raft_consensus.cc:729] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384d555300994c9797c68cad5970ebb2, State: Initialized, Role: FOLLOWER
I20260926 02:12:07.803588 6449 raft_consensus.cc:729] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef684965d8d040cdbb44d4e60d1e8df5, State: Initialized, Role: FOLLOWER
I20260926 02:12:07.804257 6448 consensus_queue.cc:260] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef684965d8d040cdbb44d4e60d1e8df5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42801 } } peers { permanent_uuid: "384d555300994c9797c68cad5970ebb2" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 34697 } } peers { permanent_uuid: "35f93f25fb614a4c870c6b04c3d90f26" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 35437 } }
I20260926 02:12:07.804258 6449 consensus_queue.cc:260] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef684965d8d040cdbb44d4e60d1e8df5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42801 } } peers { permanent_uuid: "384d555300994c9797c68cad5970ebb2" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 34697 } } peers { permanent_uuid: "35f93f25fb614a4c870c6b04c3d90f26" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 35437 } }
I20260926 02:12:07.806118 6449 ts_tablet_manager.cc:1428] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 02:12:07.806357 6448 ts_tablet_manager.cc:1428] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260926 02:12:07.806900 6429 heartbeater.cc:503] Master 127.18.159.190:39369 was elected leader, sending a full tablet report...
I20260926 02:12:07.831418 6454 raft_consensus.cc:482] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:12:07.831736 6454 raft_consensus.cc:504] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef684965d8d040cdbb44d4e60d1e8df5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42801 } } peers { permanent_uuid: "384d555300994c9797c68cad5970ebb2" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 34697 } } peers { permanent_uuid: "35f93f25fb614a4c870c6b04c3d90f26" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 35437 } }
I20260926 02:12:07.833448 6454 leader_election.cc:288] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef684965d8d040cdbb44d4e60d1e8df5 (127.18.159.129:42801), 384d555300994c9797c68cad5970ebb2 (127.18.159.131:34697)
I20260926 02:12:07.841575 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:07.833954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:55755 (local address 127.18.159.131:34697)
0926 02:12:07.834159 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:07.834165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:07.835003 (+ 838us) server_negotiation.cc:407] Connection header received
0926 02:12:07.835231 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:07.835238 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:07.835340 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:07.835434 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:07.836392 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.837547 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.839644 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.840531 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.840701 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:07.840797 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:07.841144 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 02:12:07.841302 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:12:07.841595 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:07.833998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:40339 (local address 127.18.159.129:42801)
0926 02:12:07.834175 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:07.834182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:07.835013 (+ 831us) server_negotiation.cc:407] Connection header received
0926 02:12:07.835231 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:07.835238 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:07.835323 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:07.835434 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:07.836372 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.837548 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.839680 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.840551 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.840738 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:07.840815 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:07.841215 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 02:12:07.841352 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:12:07.841704 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:07.833838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:34697 (local address 127.18.159.130:55755)
0926 02:12:07.834813 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:07.834834 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:07.835034 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:07.835480 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:07.835485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:07.835523 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:07.836136 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:07.836181 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.837679 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.837686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:07.839454 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:07.839472 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.840765 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.840771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:07.840974 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:07.841002 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 02:12:07.841077 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 02:12:07.841233 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":121,"spinlock_wait_cycles":20352,"thread_start_us":534,"threads_started":1}
I20260926 02:12:07.841746 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:07.833879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:42801 (local address 127.18.159.130:40339)
0926 02:12:07.834885 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:07.834905 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:07.835034 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:07.835480 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:07.835485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:07.835520 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:07.836202 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:07.836212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.837754 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.837758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:07.839481 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:07.839495 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.840809 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.840818 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:07.841027 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:07.841051 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 02:12:07.841166 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 02:12:07.841297 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":106,"spinlock_wait_cycles":12032,"thread_start_us":512,"threads_started":1}
I20260926 02:12:07.842882 6384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892ccbc5c7854e25994d45a7e5bde0f5" candidate_uuid: "35f93f25fb614a4c870c6b04c3d90f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "384d555300994c9797c68cad5970ebb2" is_pre_election: true
I20260926 02:12:07.843008 6117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892ccbc5c7854e25994d45a7e5bde0f5" candidate_uuid: "35f93f25fb614a4c870c6b04c3d90f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef684965d8d040cdbb44d4e60d1e8df5" is_pre_election: true
I20260926 02:12:07.843434 6384 raft_consensus.cc:2454] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f93f25fb614a4c870c6b04c3d90f26 in term 0.
I20260926 02:12:07.843647 6117 raft_consensus.cc:2454] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f93f25fb614a4c870c6b04c3d90f26 in term 0.
I20260926 02:12:07.844205 6185 leader_election.cc:302] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35f93f25fb614a4c870c6b04c3d90f26, 384d555300994c9797c68cad5970ebb2; no voters:
I20260926 02:12:07.844655 6454 raft_consensus.cc:2781] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:12:07.844821 6454 raft_consensus.cc:482] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:12:07.844934 6454 raft_consensus.cc:3037] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 [term 0 FOLLOWER]: Advancing to term 1
W20260926 02:12:07.847294 6163 tablet.cc:2369] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:12:07.847267 6454 raft_consensus.cc:504] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef684965d8d040cdbb44d4e60d1e8df5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42801 } } peers { permanent_uuid: "384d555300994c9797c68cad5970ebb2" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 34697 } } peers { permanent_uuid: "35f93f25fb614a4c870c6b04c3d90f26" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 35437 } }
I20260926 02:12:07.848017 6454 leader_election.cc:288] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 [CANDIDATE]: Term 1 election: Requested vote from peers ef684965d8d040cdbb44d4e60d1e8df5 (127.18.159.129:42801), 384d555300994c9797c68cad5970ebb2 (127.18.159.131:34697)
I20260926 02:12:07.848502 6117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892ccbc5c7854e25994d45a7e5bde0f5" candidate_uuid: "35f93f25fb614a4c870c6b04c3d90f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef684965d8d040cdbb44d4e60d1e8df5"
I20260926 02:12:07.848552 6384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892ccbc5c7854e25994d45a7e5bde0f5" candidate_uuid: "35f93f25fb614a4c870c6b04c3d90f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "384d555300994c9797c68cad5970ebb2"
I20260926 02:12:07.848699 6117 raft_consensus.cc:3037] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:07.848839 6384 raft_consensus.cc:3037] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:07.851089 6117 raft_consensus.cc:2454] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f93f25fb614a4c870c6b04c3d90f26 in term 1.
I20260926 02:12:07.851233 6384 raft_consensus.cc:2454] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f93f25fb614a4c870c6b04c3d90f26 in term 1.
I20260926 02:12:07.851583 6186 leader_election.cc:302] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35f93f25fb614a4c870c6b04c3d90f26, ef684965d8d040cdbb44d4e60d1e8df5; no voters:
I20260926 02:12:07.851958 6454 raft_consensus.cc:2781] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:07.852499 6454 raft_consensus.cc:686] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 [term 1 LEADER]: Becoming Leader. State: Replica: 35f93f25fb614a4c870c6b04c3d90f26, State: Running, Role: LEADER
I20260926 02:12:07.852984 6454 consensus_queue.cc:237] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef684965d8d040cdbb44d4e60d1e8df5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42801 } } peers { permanent_uuid: "384d555300994c9797c68cad5970ebb2" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 34697 } } peers { permanent_uuid: "35f93f25fb614a4c870c6b04c3d90f26" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 35437 } }
I20260926 02:12:07.857662 5970 catalog_manager.cc:5495] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35f93f25fb614a4c870c6b04c3d90f26 (127.18.159.130). New cstate: current_term: 1 leader_uuid: "35f93f25fb614a4c870c6b04c3d90f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef684965d8d040cdbb44d4e60d1e8df5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "384d555300994c9797c68cad5970ebb2" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 34697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35f93f25fb614a4c870c6b04c3d90f26" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 35437 } health_report { overall_health: HEALTHY } } }
W20260926 02:12:07.869782 6297 tablet.cc:2369] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:12:07.893412 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:07.888065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49172 (local address 127.18.159.130:35437)
0926 02:12:07.888307 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:07.888311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:07.888327 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:12:07.888413 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:07.888416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:07.888510 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:07.888633 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:07.888962 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.889709 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.891037 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.891420 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.891519 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:07.892049 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:07.892806 (+ 757us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:07.892940 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:07.893044 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 02:12:07.893129 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:12:07.902853 6384 raft_consensus.cc:1261] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2 [term 1 FOLLOWER]: Refusing update from remote peer 35f93f25fb614a4c870c6b04c3d90f26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:12:07.902853 6117 raft_consensus.cc:1261] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5 [term 1 FOLLOWER]: Refusing update from remote peer 35f93f25fb614a4c870c6b04c3d90f26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:12:07.903952 6459 consensus_queue.cc:1035] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef684965d8d040cdbb44d4e60d1e8df5" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 42801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:12:07.904392 6454 consensus_queue.cc:1035] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "384d555300994c9797c68cad5970ebb2" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 02:12:07.914201 6430 tablet.cc:2369] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:12:07.942000 5970 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:12:07.942337 5970 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:12:07.946208 5970 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 050625ce34e4496f88cbf254a6258a14: Sending DeleteTablet for 3 replicas of tablet 892ccbc5c7854e25994d45a7e5bde0f5
I20260926 02:12:07.947484 6096 tablet_service.cc:1514] Processing DeleteTablet for tablet 892ccbc5c7854e25994d45a7e5bde0f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:12:07 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:51248
I20260926 02:12:07.947643 6364 tablet_service.cc:1514] Processing DeleteTablet for tablet 892ccbc5c7854e25994d45a7e5bde0f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:12:07 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:59566
I20260926 02:12:07.948503 6472 tablet_replica.cc:330] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5: stopping tablet replica
I20260926 02:12:07.949059 6472 raft_consensus.cc:2229] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:12:07.949158 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 6033
I20260926 02:12:07.949174 6231 tablet_service.cc:1514] Processing DeleteTablet for tablet 892ccbc5c7854e25994d45a7e5bde0f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:12:07 UTC) from {username='kudu', principal='kudu/127.18.159.190@KRBTEST.COM'} at 127.0.0.1:49158
I20260926 02:12:07.949662 6472 raft_consensus.cc:2258] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:12:07.953405 6472 ts_tablet_manager.cc:1905] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:12:07.962642 6472 ts_tablet_manager.cc:1918] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:12:07.962857 6472 log.cc:1196] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/892ccbc5c7854e25994d45a7e5bde0f5
I20260926 02:12:07.963430 6472 ts_tablet_manager.cc:1939] T 892ccbc5c7854e25994d45a7e5bde0f5 P ef684965d8d040cdbb44d4e60d1e8df5: Deleting consensus metadata
I20260926 02:12:07.965628 5957 catalog_manager.cc:4841] TS ef684965d8d040cdbb44d4e60d1e8df5 (127.18.159.129:42801): tablet 892ccbc5c7854e25994d45a7e5bde0f5 (table test-table [id=a8bfc94a89f64adbb0873fcba5cf4ff7]) successfully deleted
I20260926 02:12:07.969169 6474 tablet_replica.cc:330] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26: stopping tablet replica
I20260926 02:12:07.969827 6474 raft_consensus.cc:2229] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:12:07.970542 6473 tablet_replica.cc:330] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2: stopping tablet replica
I20260926 02:12:07.970978 6473 raft_consensus.cc:2229] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:12:07.971536 6473 raft_consensus.cc:2258] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 26 02:12:07 dist-test-slave-03rn krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.129@KRBTEST.COM
I20260926 02:12:07.972528 6474 raft_consensus.cc:2258] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:12:07.974905 6474 ts_tablet_manager.cc:1905] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:12:07.975095 6473 ts_tablet_manager.cc:1905] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:12:07.982328 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:07.951003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51264 (local address 127.18.159.129:42801)
0926 02:12:07.951218 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:07.951224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:07.960922 (+ 9698us) server_negotiation.cc:407] Connection header received
0926 02:12:07.961377 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:07.961384 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:07.961468 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:07.961539 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:07.963820 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.964983 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.967135 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:07.967561 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:07.967728 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:07.973521 (+ 5793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:07.973544 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:07.973563 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:07.973620 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:07.977621 (+ 4001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:07.978692 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:07.978704 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:07.978707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:07.978912 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:07.979511 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:07.979517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:07.979520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:07.980039 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:07.980363 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:07.981675 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0926 02:12:07.981938 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:12:07.982374 6473 ts_tablet_manager.cc:1918] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:12:07.982540 6473 log.cc:1196] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/892ccbc5c7854e25994d45a7e5bde0f5
I20260926 02:12:07.982945 6474 ts_tablet_manager.cc:1918] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:12:07.983119 6473 ts_tablet_manager.cc:1939] T 892ccbc5c7854e25994d45a7e5bde0f5 P 384d555300994c9797c68cad5970ebb2: Deleting consensus metadata
I20260926 02:12:07.983137 6474 log.cc:1196] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/892ccbc5c7854e25994d45a7e5bde0f5
I20260926 02:12:07.983563 6157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:07.983763 6474 ts_tablet_manager.cc:1939] T 892ccbc5c7854e25994d45a7e5bde0f5 P 35f93f25fb614a4c870c6b04c3d90f26: Deleting consensus metadata
I20260926 02:12:07.985311 5956 catalog_manager.cc:4841] TS 35f93f25fb614a4c870c6b04c3d90f26 (127.18.159.130:35437): tablet 892ccbc5c7854e25994d45a7e5bde0f5 (table test-table [id=a8bfc94a89f64adbb0873fcba5cf4ff7]) successfully deleted
I20260926 02:12:07.985615 5956 catalog_manager.cc:4841] TS 384d555300994c9797c68cad5970ebb2 (127.18.159.131:34697): tablet 892ccbc5c7854e25994d45a7e5bde0f5 (table test-table [id=a8bfc94a89f64adbb0873fcba5cf4ff7]) successfully deleted
I20260926 02:12:08.084051 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 6033
I20260926 02:12:08.102376 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 6168
Sep 26 02:12:08 dist-test-slave-03rn krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.130@KRBTEST.COM
I20260926 02:12:08.120687 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:08.103232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49188 (local address 127.18.159.130:35437)
0926 02:12:08.103429 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:08.103433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:08.103451 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:12:08.103655 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:08.103659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:08.103726 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:08.103825 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:08.105579 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:08.106429 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:08.108463 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:08.108910 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:08.109027 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:08.114200 (+ 5173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:08.114217 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:08.114232 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:08.114280 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:08.117641 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:08.118414 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:08.118421 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:08.118424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:08.118581 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:08.118985 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:08.118996 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:08.118998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:08.119369 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:08.119736 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:08.120027 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 02:12:08.120301 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 02:12:08.121460 6291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:08.238957 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 6168
I20260926 02:12:08.258396 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 6301
Sep 26 02:12:08 dist-test-slave-03rn krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.131@KRBTEST.COM
I20260926 02:12:08.275898 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:08.259370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59570 (local address 127.18.159.131:34697)
0926 02:12:08.259575 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:08.259583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:08.259686 (+ 103us) server_negotiation.cc:407] Connection header received
0926 02:12:08.259903 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:08.259913 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:08.260002 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:08.260121 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:08.261703 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:08.262487 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:08.263943 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:08.264484 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:08.264656 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:08.269568 (+ 4912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:08.269586 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:08.269603 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:08.269647 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:08.272778 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:08.273653 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:08.273661 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:08.273663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:08.273829 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:08.274283 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:08.274291 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:08.274299 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:08.274671 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:08.274996 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:08.275323 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 02:12:08.275557 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:12:08.276643 6424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:08.367208 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 6301
I20260926 02:12:08.387657 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 5939
I20260926 02:12:08.388808 6002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:08.500772 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 5939
2026-09-26T02:12:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4459 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T02:12:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:08Z Disabled control of system clock
I20260926 02:12:08.578995 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:45967
--webserver_interface=127.18.159.190
--webserver_port=0
--builtin_ntp_servers=127.18.159.148:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:45967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:12:08.930132 6483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:08.930472 6483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:08.930562 6483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:08.938222 6483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:08.938355 6483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:08.938395 6483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:08.938454 6483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:08.938491 6483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:08.948045 6483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:45967
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:45967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:08.949414 6483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:08.951246 6483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:08.964751 6491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:08.964702 6489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:08.964798 6488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:08.965842 6483 server_base.cc:1034] running on GCE node
I20260926 02:12:08.966696 6483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:08.968345 6483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:08.969587 6483 hybrid_clock.cc:648] HybridClock initialized: now 1790388728969519 us; error 67 us; skew 500 ppm
I20260926 02:12:08.970021 6483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:12:08.972960 6483 webserver.cc:460] Webserver started at http://127.18.159.190:43381/ using document root <none> and password file <none>
I20260926 02:12:08.973753 6483 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:08.973888 6483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:08.974247 6483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:08.977499 6483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "5c2bef36a0f54760a7ba776549f73b7e"
format_stamp: "Formatted at 2026-09-26 02:12:08 on dist-test-slave-03rn"
server_key: "3db12ce6ef5f0ac658f851901ebac08e"
server_key_iv: "41eb6fab74c6be68a7855fac443fc5eb"
server_key_version: "encryptionkey@0"
I20260926 02:12:08.978402 6483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "5c2bef36a0f54760a7ba776549f73b7e"
format_stamp: "Formatted at 2026-09-26 02:12:08 on dist-test-slave-03rn"
server_key: "3db12ce6ef5f0ac658f851901ebac08e"
server_key_iv: "41eb6fab74c6be68a7855fac443fc5eb"
server_key_version: "encryptionkey@0"
I20260926 02:12:08.984050 6483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:12:08.987939 6497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:08.989588 6483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:12:08.989820 6483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "5c2bef36a0f54760a7ba776549f73b7e"
format_stamp: "Formatted at 2026-09-26 02:12:08 on dist-test-slave-03rn"
server_key: "3db12ce6ef5f0ac658f851901ebac08e"
server_key_iv: "41eb6fab74c6be68a7855fac443fc5eb"
server_key_version: "encryptionkey@0"
I20260926 02:12:08.990087 6483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:09.013783 6483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:09.014739 6483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:09.015066 6483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:09.035423 6483 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:45967
I20260926 02:12:09.035449 6548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:45967 every 8 connection(s)
I20260926 02:12:09.037020 6483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:09.041095 6549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:09.045603 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 6483
I20260926 02:12:09.045809 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:09.046304 19070 external_mini_cluster.cc:1428] Setting key 179b06ccc57520ec72d27bba3490eaa4
I20260926 02:12:09.054673 6549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e: Bootstrap starting.
I20260926 02:12:09.058630 6549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:09.060060 6549 log.cc:824] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:09.063681 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:09.048468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40048 (local address 127.18.159.190:45967)
0926 02:12:09.049669 (+ 1201us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:09.049684 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:09.049751 (+ 67us) server_negotiation.cc:407] Connection header received
0926 02:12:09.052606 (+ 2855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:09.052642 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:09.053354 (+ 712us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:09.053845 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:09.054456 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:09.056525 (+ 2069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:09.058065 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:09.059287 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:09.059481 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:09.060770 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:09.060785 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:09.060808 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:09.061430 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:09.061571 (+ 141us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:09.061599 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:09.061720 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:09.061973 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 02:12:09.062325 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":384,"threads_started":1}
I20260926 02:12:09.063836 6549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e: No bootstrap required, opened a new log
I20260926 02:12:09.072252 6549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c2bef36a0f54760a7ba776549f73b7e" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45967 } }
I20260926 02:12:09.072781 6549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:09.072893 6549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c2bef36a0f54760a7ba776549f73b7e, State: Initialized, Role: FOLLOWER
I20260926 02:12:09.073529 6549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c2bef36a0f54760a7ba776549f73b7e" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45967 } }
I20260926 02:12:09.073776 6549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:12:09.073941 6549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:12:09.074128 6549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:09.076520 6549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c2bef36a0f54760a7ba776549f73b7e" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45967 } }
I20260926 02:12:09.077085 6549 leader_election.cc:302] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c2bef36a0f54760a7ba776549f73b7e; no voters:
I20260926 02:12:09.077754 6549 leader_election.cc:288] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:12:09.078013 6554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:09.078833 6554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e [term 1 LEADER]: Becoming Leader. State: Replica: 5c2bef36a0f54760a7ba776549f73b7e, State: Running, Role: LEADER
I20260926 02:12:09.079509 6554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c2bef36a0f54760a7ba776549f73b7e" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45967 } }
I20260926 02:12:09.080123 6549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:09.082723 6555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c2bef36a0f54760a7ba776549f73b7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c2bef36a0f54760a7ba776549f73b7e" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45967 } } }
I20260926 02:12:09.083958 6555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:09.083716 6556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c2bef36a0f54760a7ba776549f73b7e. Latest consensus state: current_term: 1 leader_uuid: "5c2bef36a0f54760a7ba776549f73b7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c2bef36a0f54760a7ba776549f73b7e" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 45967 } } }
I20260926 02:12:09.084218 6556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:09.085435 6563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:09.097617 6563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:09.107704 6563 catalog_manager.cc:1347] Generated new cluster ID: 30c0b8e989e14c19a3c74d73d67f1d3a
I20260926 02:12:09.107838 6563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:09.134187 6563 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:09.135679 6563 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:09.158531 6563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e: Generated new TSK 0
I20260926 02:12:09.159451 6563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:12:09.169785 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.129:0
--local_ip_for_outbound_sockets=127.18.159.129
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:45967
--builtin_ntp_servers=127.18.159.148:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:12:09.561483 6573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:09.561898 6573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:09.561996 6573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:09.569599 6573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:09.569715 6573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:09.569823 6573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.129
I20260926 02:12:09.579238 6573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.159.129
--webserver_port=0
--tserver_master_addrs=127.18.159.190:45967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.129
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:09.580629 6573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:09.582561 6573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:09.596724 6578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:09.596763 6579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:09.596678 6581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:09.601930 6573 server_base.cc:1034] running on GCE node
I20260926 02:12:09.602663 6573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:09.603719 6573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:09.604967 6573 hybrid_clock.cc:648] HybridClock initialized: now 1790388729604920 us; error 66 us; skew 500 ppm
I20260926 02:12:09.605321 6573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:12:09.607781 6573 webserver.cc:460] Webserver started at http://127.18.159.129:45283/ using document root <none> and password file <none>
I20260926 02:12:09.608600 6573 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:09.608734 6573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:09.609064 6573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:09.612234 6573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-0/data/instance:
uuid: "79b87c4100924c70960523a73c3f6257"
format_stamp: "Formatted at 2026-09-26 02:12:09 on dist-test-slave-03rn"
server_key: "ed2f9b9b663c4f14bed0513d0731e301"
server_key_iv: "d5d447b041508fdec08d5cd48dce270d"
server_key_version: "encryptionkey@0"
I20260926 02:12:09.613091 6573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance:
uuid: "79b87c4100924c70960523a73c3f6257"
format_stamp: "Formatted at 2026-09-26 02:12:09 on dist-test-slave-03rn"
server_key: "ed2f9b9b663c4f14bed0513d0731e301"
server_key_iv: "d5d447b041508fdec08d5cd48dce270d"
server_key_version: "encryptionkey@0"
I20260926 02:12:09.618623 6573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:12:09.622558 6587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:09.623977 6573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:12:09.624250 6573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-0/wal
uuid: "79b87c4100924c70960523a73c3f6257"
format_stamp: "Formatted at 2026-09-26 02:12:09 on dist-test-slave-03rn"
server_key: "ed2f9b9b663c4f14bed0513d0731e301"
server_key_iv: "d5d447b041508fdec08d5cd48dce270d"
server_key_version: "encryptionkey@0"
I20260926 02:12:09.624507 6573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:09.644869 6573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:09.645829 6573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:09.646176 6573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:09.647250 6573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:12:09.648953 6573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:12:09.649101 6573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:09.649247 6573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:12:09.649344 6573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:09.685485 6573 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.129:33671
I20260926 02:12:09.685592 6700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.129:33671 every 8 connection(s)
I20260926 02:12:09.686981 6573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-0/data/info.pb
I20260926 02:12:09.691159 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 6573
I20260926 02:12:09.691340 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-0/wal/instance
I20260926 02:12:09.691869 19070 external_mini_cluster.cc:1428] Setting key c705b1b14c16653e94fa7b172d1bc92b
I20260926 02:12:09.695333 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.130:0
--local_ip_for_outbound_sockets=127.18.159.130
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:45967
--builtin_ntp_servers=127.18.159.148:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:12:09.702824 6703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:09.689748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.129:48401 (local address 127.18.159.190:45967)
0926 02:12:09.690206 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:09.690212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:09.691819 (+ 1607us) server_negotiation.cc:407] Connection header received
0926 02:12:09.694185 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:09.694190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:09.694257 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:09.694352 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:09.696147 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:09.696970 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:09.699630 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:09.700042 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:09.700141 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:09.702001 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:09.702006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:09.702009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:09.702030 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:09.702081 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:09.702095 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:09.702175 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:09.702463 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 02:12:09.702554 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":219,"threads_started":1}
I20260926 02:12:09.704492 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:09.690440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:45967 (local address 127.18.159.129:48401)
0926 02:12:09.691600 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:09.691653 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:09.693759 (+ 2106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:09.694675 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:09.694687 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:09.694824 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:09.695927 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:09.695960 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:09.697156 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:09.697164 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:09.699461 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:09.699481 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:09.700330 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:09.700340 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:09.700714 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:09.701570 (+ 856us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:12:09.701577 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:09.701635 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:12:09.701758 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:09.702203 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:09.702217 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:09.702229 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:12:09.702442 (+ 213us) client_negotiation.cc:240] Negotiation successful
0926 02:12:09.702915 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":325,"threads_started":1}
I20260926 02:12:09.706451 6701 heartbeater.cc:347] Connected to a master server at 127.18.159.190:45967
I20260926 02:12:09.706812 6701 heartbeater.cc:464] Registering TS with master...
I20260926 02:12:09.707672 6701 heartbeater.cc:511] Master 127.18.159.190:45967 requested a full tablet report, sending...
I20260926 02:12:09.710125 6514 ts_manager.cc:194] Registered new tserver with Master: 79b87c4100924c70960523a73c3f6257 (127.18.159.129:33671)
I20260926 02:12:09.711675 6514 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.159.129:48401
W20260926 02:12:10.043617 6705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:10.043946 6705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:10.044101 6705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:10.051734 6705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:10.051914 6705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:10.052095 6705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.130
I20260926 02:12:10.061744 6705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.159.130
--webserver_port=0
--tserver_master_addrs=127.18.159.190:45967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.130
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:10.063135 6705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:10.065109 6705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:10.079470 6710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:10.079638 6711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:10.079653 6713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:10.080965 6705 server_base.cc:1034] running on GCE node
I20260926 02:12:10.081848 6705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:10.083015 6705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:10.084303 6705 hybrid_clock.cc:648] HybridClock initialized: now 1790388730084239 us; error 45 us; skew 500 ppm
I20260926 02:12:10.084731 6705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:12:10.087564 6705 webserver.cc:460] Webserver started at http://127.18.159.130:35547/ using document root <none> and password file <none>
I20260926 02:12:10.088393 6705 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:10.088500 6705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:10.088792 6705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:10.092098 6705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-1/data/instance:
uuid: "ed8206169bc84595a5769a74a8a3716d"
format_stamp: "Formatted at 2026-09-26 02:12:10 on dist-test-slave-03rn"
server_key: "131edd09dd0974803e919078e178bd0b"
server_key_iv: "735ccc63cbfd0a73ece96a998ee5aa57"
server_key_version: "encryptionkey@0"
I20260926 02:12:10.093161 6705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance:
uuid: "ed8206169bc84595a5769a74a8a3716d"
format_stamp: "Formatted at 2026-09-26 02:12:10 on dist-test-slave-03rn"
server_key: "131edd09dd0974803e919078e178bd0b"
server_key_iv: "735ccc63cbfd0a73ece96a998ee5aa57"
server_key_version: "encryptionkey@0"
I20260926 02:12:10.099473 6705 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 02:12:10.103627 6719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:10.105347 6705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:12:10.105664 6705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-1/wal
uuid: "ed8206169bc84595a5769a74a8a3716d"
format_stamp: "Formatted at 2026-09-26 02:12:10 on dist-test-slave-03rn"
server_key: "131edd09dd0974803e919078e178bd0b"
server_key_iv: "735ccc63cbfd0a73ece96a998ee5aa57"
server_key_version: "encryptionkey@0"
I20260926 02:12:10.105912 6705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:10.126143 6705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:10.127305 6705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:10.127682 6705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:10.128886 6705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:12:10.130530 6705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:12:10.130628 6705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:10.130781 6705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:12:10.130879 6705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:10.166836 6705 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.130:43589
I20260926 02:12:10.166895 6832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.130:43589 every 8 connection(s)
I20260926 02:12:10.168303 6705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-1/data/info.pb
I20260926 02:12:10.174731 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 6705
I20260926 02:12:10.174939 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-1/wal/instance
I20260926 02:12:10.175483 19070 external_mini_cluster.cc:1428] Setting key 3934f723f7235eaa14bbba52cb529721
I20260926 02:12:10.178865 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.159.131:0
--local_ip_for_outbound_sockets=127.18.159.131
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:45967
--builtin_ntp_servers=127.18.159.148:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:12:10.183553 6703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:10.170838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.130:39495 (local address 127.18.159.190:45967)
0926 02:12:10.171027 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:10.171032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:10.172653 (+ 1621us) server_negotiation.cc:407] Connection header received
0926 02:12:10.174613 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:10.174618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:10.174693 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:10.174792 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:10.176426 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.177472 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.180134 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.180665 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.180818 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:10.182592 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:10.182600 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:10.182607 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:10.182635 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:10.182676 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:10.182692 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:10.182774 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:10.183054 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 02:12:10.183168 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:12:10.185122 6835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:10.171420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:45967 (local address 127.18.159.130:39495)
0926 02:12:10.172481 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:10.172519 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:10.174218 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:10.175126 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:10.175139 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:10.175275 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:10.176206 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:10.176240 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.177645 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.177650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:10.179935 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:10.179958 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.180936 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.180946 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:10.181318 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:10.182165 (+ 847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:12:10.182170 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:10.182230 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:12:10.182354 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:10.182800 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:10.182813 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:10.182833 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 02:12:10.183034 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 02:12:10.183512 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":287,"threads_started":1}
I20260926 02:12:10.186790 6833 heartbeater.cc:347] Connected to a master server at 127.18.159.190:45967
I20260926 02:12:10.187129 6833 heartbeater.cc:464] Registering TS with master...
I20260926 02:12:10.188074 6833 heartbeater.cc:511] Master 127.18.159.190:45967 requested a full tablet report, sending...
I20260926 02:12:10.189893 6514 ts_manager.cc:194] Registered new tserver with Master: ed8206169bc84595a5769a74a8a3716d (127.18.159.130:43589)
I20260926 02:12:10.190707 6514 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.159.130:39495
W20260926 02:12:10.523321 6836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:10.523694 6836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:10.523851 6836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:10.532676 6836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:10.532881 6836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:10.533074 6836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.159.131
I20260926 02:12:10.542665 6836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.159.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.159.131
--webserver_port=0
--tserver_master_addrs=127.18.159.190:45967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.159.131
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:10.544030 6836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:10.545853 6836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:10.559118 6844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:10.559242 6841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:10.559222 6842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:10.560461 6836 server_base.cc:1034] running on GCE node
I20260926 02:12:10.561173 6836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:10.562175 6836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:10.563419 6836 hybrid_clock.cc:648] HybridClock initialized: now 1790388730563365 us; error 50 us; skew 500 ppm
I20260926 02:12:10.563762 6836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:12:10.566263 6836 webserver.cc:460] Webserver started at http://127.18.159.131:37233/ using document root <none> and password file <none>
I20260926 02:12:10.567013 6836 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:10.567142 6836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:10.567461 6836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:10.570475 6836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-2/data/instance:
uuid: "2450ada3a46d4e108e5bf86bd674a231"
format_stamp: "Formatted at 2026-09-26 02:12:10 on dist-test-slave-03rn"
server_key: "b277c2889ea3994ec601ba095bc9f80f"
server_key_iv: "064d60e2257106adc676c096cc65c7fe"
server_key_version: "encryptionkey@0"
I20260926 02:12:10.571273 6836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance:
uuid: "2450ada3a46d4e108e5bf86bd674a231"
format_stamp: "Formatted at 2026-09-26 02:12:10 on dist-test-slave-03rn"
server_key: "b277c2889ea3994ec601ba095bc9f80f"
server_key_iv: "064d60e2257106adc676c096cc65c7fe"
server_key_version: "encryptionkey@0"
I20260926 02:12:10.576603 6836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:12:10.580108 6850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:10.581722 6836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:12:10.581864 6836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-2/wal
uuid: "2450ada3a46d4e108e5bf86bd674a231"
format_stamp: "Formatted at 2026-09-26 02:12:10 on dist-test-slave-03rn"
server_key: "b277c2889ea3994ec601ba095bc9f80f"
server_key_iv: "064d60e2257106adc676c096cc65c7fe"
server_key_version: "encryptionkey@0"
I20260926 02:12:10.582049 6836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:10.600144 6836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:10.601109 6836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:10.601430 6836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:10.602444 6836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:12:10.603919 6836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:12:10.604043 6836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:10.604205 6836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:12:10.604297 6836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 02:12:10.637848 6836 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.131:35795
I20260926 02:12:10.637907 6963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.131:35795 every 8 connection(s)
I20260926 02:12:10.639235 6836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-2/data/info.pb
I20260926 02:12:10.647069 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 6836
I20260926 02:12:10.647254 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-2/wal/instance
I20260926 02:12:10.647691 19070 external_mini_cluster.cc:1428] Setting key 985de8a2b489b364ec2b902371e3d225
I20260926 02:12:10.652822 6703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:10.641796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:51129 (local address 127.18.159.190:45967)
0926 02:12:10.641979 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:10.641984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:10.643498 (+ 1514us) server_negotiation.cc:407] Connection header received
0926 02:12:10.645458 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:10.645464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:10.645565 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:10.645686 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:10.646867 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.647744 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.649995 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.650626 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.650747 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:10.652109 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:10.652114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:10.652117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:10.652134 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:10.652211 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:10.652233 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:10.652304 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:10.652510 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 02:12:10.652585 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:12:10.653892 6966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:10.642331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:45967 (local address 127.18.159.131:51129)
0926 02:12:10.643260 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:10.643339 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:10.645147 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:10.645939 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:10.645954 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:10.646049 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:10.646635 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:10.646682 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.647928 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.647934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:10.649834 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:10.649846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.650864 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.650869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:10.651096 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:10.651789 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:12:10.651792 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:10.651832 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:12:10.651915 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:10.652316 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:10.652324 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:10.652330 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 02:12:10.652483 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 02:12:10.652802 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":270,"threads_started":1}
I20260926 02:12:10.655628 6964 heartbeater.cc:347] Connected to a master server at 127.18.159.190:45967
I20260926 02:12:10.655928 6964 heartbeater.cc:464] Registering TS with master...
I20260926 02:12:10.656771 6964 heartbeater.cc:511] Master 127.18.159.190:45967 requested a full tablet report, sending...
I20260926 02:12:10.658258 6514 ts_manager.cc:194] Registered new tserver with Master: 2450ada3a46d4e108e5bf86bd674a231 (127.18.159.131:35795)
I20260926 02:12:10.659080 6514 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.159.131:51129
I20260926 02:12:10.663249 19070 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:12:10.673895 6703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:10.667640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52898 (local address 127.18.159.190:45967)
0926 02:12:10.667924 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:10.667931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:10.668275 (+ 344us) server_negotiation.cc:407] Connection header received
0926 02:12:10.668493 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:10.668497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:10.668563 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:10.668678 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:10.669251 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.669959 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.671362 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.671804 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.671941 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:10.673107 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:10.673112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:10.673115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:10.673133 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:10.673178 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:10.673190 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:10.673300 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:10.673451 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 02:12:10.673547 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 02:12:10.678059 6514 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:12:10.685832 6514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:12:10.715058 6701 heartbeater.cc:503] Master 127.18.159.190:45967 was elected leader, sending a full tablet report...
I20260926 02:12:10.724288 6978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:10.700351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.131:35795 (local address 127.0.0.1:59600)
0926 02:12:10.715006 (+ 14655us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:10.715032 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:10.715209 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:10.716572 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:10.716579 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:10.716613 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:10.717077 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:10.717092 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.719058 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.719063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:10.721375 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:10.721387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.723090 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.723097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:10.723343 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:10.723417 (+ 74us) client_negotiation.cc:769] Sending connection context
0926 02:12:10.723561 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 02:12:10.723769 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14454,"mutex_wait_us":34,"thread_start_us":14268,"threads_started":1}
I20260926 02:12:10.725414 6979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:10.700844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59600 (local address 127.18.159.131:35795)
0926 02:12:10.714425 (+ 13581us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:10.714438 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:10.715399 (+ 961us) server_negotiation.cc:407] Connection header received
0926 02:12:10.715545 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:10.715559 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:10.715929 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:10.716198 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:10.717344 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.718885 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.721581 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.722868 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.723104 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:10.723346 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:10.724836 (+ 1490us) server_negotiation.cc:299] Negotiation successful
0926 02:12:10.725021 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13311,"thread_start_us":13129,"threads_started":1}
I20260926 02:12:10.726642 6976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:10.700086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:33671 (local address 127.0.0.1:37432)
0926 02:12:10.712601 (+ 12515us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:10.712656 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:10.712859 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:10.717492 (+ 4633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:10.717500 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:10.717541 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:10.718082 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:10.718099 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.721153 (+ 3054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.721159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:10.723371 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:10.723388 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.725822 (+ 2434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.725829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:10.726012 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:10.726046 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 02:12:10.726136 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 02:12:10.726279 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12148,"mutex_wait_us":3,"thread_start_us":12984,"threads_started":1}
I20260926 02:12:10.727264 6977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:10.700179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37432 (local address 127.18.159.129:33671)
0926 02:12:10.716693 (+ 16514us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:10.716719 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:10.716757 (+ 38us) server_negotiation.cc:407] Connection header received
0926 02:12:10.716878 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:10.716889 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:10.717197 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:10.718346 (+ 1149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:10.718645 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.720981 (+ 2336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.723591 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.725201 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.725412 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:10.725663 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:10.726234 (+ 571us) server_negotiation.cc:299] Negotiation successful
0926 02:12:10.726438 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16282,"thread_start_us":16143,"threads_started":1}
I20260926 02:12:10.729478 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:10.716172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:43589 (local address 127.0.0.1:43360)
0926 02:12:10.720865 (+ 4693us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:10.720895 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:10.721031 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:10.721684 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:10.721688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:10.721716 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:10.722161 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:10.722174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.723971 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.723976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:10.726322 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:10.726336 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.728553 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.728560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:10.728744 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:10.728791 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 02:12:10.728899 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 02:12:10.729044 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4492,"thread_start_us":4333,"threads_started":1}
I20260926 02:12:10.729494 6980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:10.715431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43360 (local address 127.18.159.130:43589)
0926 02:12:10.716091 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:10.716102 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:10.721035 (+ 4933us) server_negotiation.cc:407] Connection header received
0926 02:12:10.721158 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:10.721167 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:10.721471 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:10.721685 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:10.722372 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.723785 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.726571 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.728064 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.728225 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:10.728453 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:10.728976 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 02:12:10.729144 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":381,"threads_started":1}
I20260926 02:12:10.729789 6899 tablet_service.cc:1467] Processing CreateTablet for tablet a705fe2381244c6c80b416d2fdf11a1f (DEFAULT_TABLE table=test-table [id=dab25ef006d34646be8d60a9f9c63bbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:12:10.731362 6899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a705fe2381244c6c80b416d2fdf11a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:10.731042 6636 tablet_service.cc:1467] Processing CreateTablet for tablet a705fe2381244c6c80b416d2fdf11a1f (DEFAULT_TABLE table=test-table [id=dab25ef006d34646be8d60a9f9c63bbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:12:10.732566 6636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a705fe2381244c6c80b416d2fdf11a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:10.733569 6768 tablet_service.cc:1467] Processing CreateTablet for tablet a705fe2381244c6c80b416d2fdf11a1f (DEFAULT_TABLE table=test-table [id=dab25ef006d34646be8d60a9f9c63bbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:12:10.734932 6768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a705fe2381244c6c80b416d2fdf11a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:10.743202 6982 tablet_bootstrap.cc:492] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231: Bootstrap starting.
I20260926 02:12:10.746605 6982 tablet_bootstrap.cc:654] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:10.747859 6983 tablet_bootstrap.cc:492] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257: Bootstrap starting.
I20260926 02:12:10.747982 6982 log.cc:824] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:10.749866 6985 tablet_bootstrap.cc:492] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d: Bootstrap starting.
I20260926 02:12:10.752616 6983 tablet_bootstrap.cc:654] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:10.754031 6985 tablet_bootstrap.cc:654] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:10.754041 6983 log.cc:824] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:10.754138 6982 tablet_bootstrap.cc:492] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231: No bootstrap required, opened a new log
I20260926 02:12:10.754448 6982 ts_tablet_manager.cc:1397] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 02:12:10.755092 6985 log.cc:824] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:10.758054 6983 tablet_bootstrap.cc:492] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257: No bootstrap required, opened a new log
I20260926 02:12:10.758324 6985 tablet_bootstrap.cc:492] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d: No bootstrap required, opened a new log
I20260926 02:12:10.758370 6983 ts_tablet_manager.cc:1397] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 02:12:10.758596 6985 ts_tablet_manager.cc:1397] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 02:12:10.765914 6982 raft_consensus.cc:348] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b87c4100924c70960523a73c3f6257" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33671 } } peers { permanent_uuid: "2450ada3a46d4e108e5bf86bd674a231" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35795 } } peers { permanent_uuid: "ed8206169bc84595a5769a74a8a3716d" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 43589 } }
I20260926 02:12:10.766449 6982 raft_consensus.cc:374] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:10.766542 6982 raft_consensus.cc:729] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2450ada3a46d4e108e5bf86bd674a231, State: Initialized, Role: FOLLOWER
I20260926 02:12:10.766330 6983 raft_consensus.cc:348] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b87c4100924c70960523a73c3f6257" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33671 } } peers { permanent_uuid: "2450ada3a46d4e108e5bf86bd674a231" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35795 } } peers { permanent_uuid: "ed8206169bc84595a5769a74a8a3716d" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 43589 } }
I20260926 02:12:10.766855 6983 raft_consensus.cc:374] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:10.766964 6983 raft_consensus.cc:729] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b87c4100924c70960523a73c3f6257, State: Initialized, Role: FOLLOWER
I20260926 02:12:10.767571 6983 consensus_queue.cc:260] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b87c4100924c70960523a73c3f6257" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33671 } } peers { permanent_uuid: "2450ada3a46d4e108e5bf86bd674a231" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35795 } } peers { permanent_uuid: "ed8206169bc84595a5769a74a8a3716d" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 43589 } }
I20260926 02:12:10.767479 6982 consensus_queue.cc:260] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b87c4100924c70960523a73c3f6257" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33671 } } peers { permanent_uuid: "2450ada3a46d4e108e5bf86bd674a231" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35795 } } peers { permanent_uuid: "ed8206169bc84595a5769a74a8a3716d" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 43589 } }
I20260926 02:12:10.769007 6964 heartbeater.cc:503] Master 127.18.159.190:45967 was elected leader, sending a full tablet report...
I20260926 02:12:10.769641 6982 ts_tablet_manager.cc:1428] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260926 02:12:10.769457 6985 raft_consensus.cc:348] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b87c4100924c70960523a73c3f6257" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33671 } } peers { permanent_uuid: "2450ada3a46d4e108e5bf86bd674a231" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35795 } } peers { permanent_uuid: "ed8206169bc84595a5769a74a8a3716d" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 43589 } }
I20260926 02:12:10.769945 6985 raft_consensus.cc:374] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:10.770053 6985 raft_consensus.cc:729] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed8206169bc84595a5769a74a8a3716d, State: Initialized, Role: FOLLOWER
I20260926 02:12:10.770678 6985 consensus_queue.cc:260] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b87c4100924c70960523a73c3f6257" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33671 } } peers { permanent_uuid: "2450ada3a46d4e108e5bf86bd674a231" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35795 } } peers { permanent_uuid: "ed8206169bc84595a5769a74a8a3716d" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 43589 } }
I20260926 02:12:10.771495 6983 ts_tablet_manager.cc:1428] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260926 02:12:10.773056 6833 heartbeater.cc:503] Master 127.18.159.190:45967 was elected leader, sending a full tablet report...
I20260926 02:12:10.773795 6985 ts_tablet_manager.cc:1428] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20260926 02:12:10.890957 6965 tablet.cc:2369] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:12:10.920501 6834 tablet.cc:2369] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:12:10.929831 6989 raft_consensus.cc:482] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:12:10.930160 6989 raft_consensus.cc:504] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b87c4100924c70960523a73c3f6257" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33671 } } peers { permanent_uuid: "2450ada3a46d4e108e5bf86bd674a231" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35795 } } peers { permanent_uuid: "ed8206169bc84595a5769a74a8a3716d" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 43589 } }
I20260926 02:12:10.932426 6989 leader_election.cc:288] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79b87c4100924c70960523a73c3f6257 (127.18.159.129:33671), ed8206169bc84595a5769a74a8a3716d (127.18.159.130:43589)
I20260926 02:12:10.938738 6977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:10.933013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:42301 (local address 127.18.159.129:33671)
0926 02:12:10.933258 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:10.933267 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:10.933295 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:12:10.933394 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:10.933399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:10.933507 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:10.933605 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:10.934149 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.935176 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.936998 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.937834 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.938022 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:10.938096 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:10.938339 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 02:12:10.938481 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 02:12:10.938786 6966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:10.932715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.129:33671 (local address 127.18.159.131:42301)
0926 02:12:10.932946 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:10.932969 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:10.933115 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:10.933782 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:10.933786 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:10.933808 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:10.933998 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:10.934007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.935311 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.935315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:10.936822 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:10.936834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.938003 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.938009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:10.938178 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:10.938201 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:12:10.938313 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 02:12:10.938467 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"spinlock_wait_cycles":18816}
I20260926 02:12:10.939015 6980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:10.933389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.159.131:60727 (local address 127.18.159.130:43589)
0926 02:12:10.933594 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:10.933601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:10.933791 (+ 190us) server_negotiation.cc:407] Connection header received
0926 02:12:10.933991 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:10.934003 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:10.934103 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:10.934197 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:12:10.934635 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.935536 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.937350 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.938096 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.938282 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:10.938348 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:10.938630 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 02:12:10.938754 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:12:10.939064 6992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:10.932831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.130:43589 (local address 127.18.159.131:60727)
0926 02:12:10.933682 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:10.933705 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:10.933851 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:10.934199 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:10.934202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:10.934218 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:12:10.934486 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:10.934492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.935693 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.935698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:10.937169 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:10.937181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:10.938293 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:10.938299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:10.938471 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:10.938494 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:12:10.938602 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 02:12:10.938729 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":542,"threads_started":1}
W20260926 02:12:10.939682 6702 tablet.cc:2369] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:12:10.940021 6656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a705fe2381244c6c80b416d2fdf11a1f" candidate_uuid: "2450ada3a46d4e108e5bf86bd674a231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b87c4100924c70960523a73c3f6257" is_pre_election: true
I20260926 02:12:10.940094 6788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a705fe2381244c6c80b416d2fdf11a1f" candidate_uuid: "2450ada3a46d4e108e5bf86bd674a231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed8206169bc84595a5769a74a8a3716d" is_pre_election: true
I20260926 02:12:10.940701 6788 raft_consensus.cc:2454] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2450ada3a46d4e108e5bf86bd674a231 in term 0.
I20260926 02:12:10.940696 6656 raft_consensus.cc:2454] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2450ada3a46d4e108e5bf86bd674a231 in term 0.
I20260926 02:12:10.941450 6854 leader_election.cc:302] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2450ada3a46d4e108e5bf86bd674a231, 79b87c4100924c70960523a73c3f6257, ed8206169bc84595a5769a74a8a3716d; no voters:
I20260926 02:12:10.941924 6989 raft_consensus.cc:2781] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:12:10.942077 6989 raft_consensus.cc:482] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:12:10.942183 6989 raft_consensus.cc:3037] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:10.944433 6989 raft_consensus.cc:504] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b87c4100924c70960523a73c3f6257" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33671 } } peers { permanent_uuid: "2450ada3a46d4e108e5bf86bd674a231" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35795 } } peers { permanent_uuid: "ed8206169bc84595a5769a74a8a3716d" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 43589 } }
I20260926 02:12:10.945062 6989 leader_election.cc:288] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 [CANDIDATE]: Term 1 election: Requested vote from peers 79b87c4100924c70960523a73c3f6257 (127.18.159.129:33671), ed8206169bc84595a5769a74a8a3716d (127.18.159.130:43589)
I20260926 02:12:10.945601 6656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a705fe2381244c6c80b416d2fdf11a1f" candidate_uuid: "2450ada3a46d4e108e5bf86bd674a231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b87c4100924c70960523a73c3f6257"
I20260926 02:12:10.945616 6788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a705fe2381244c6c80b416d2fdf11a1f" candidate_uuid: "2450ada3a46d4e108e5bf86bd674a231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed8206169bc84595a5769a74a8a3716d"
I20260926 02:12:10.945856 6656 raft_consensus.cc:3037] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:10.945874 6788 raft_consensus.cc:3037] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:10.948218 6656 raft_consensus.cc:2454] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2450ada3a46d4e108e5bf86bd674a231 in term 1.
I20260926 02:12:10.948259 6788 raft_consensus.cc:2454] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2450ada3a46d4e108e5bf86bd674a231 in term 1.
I20260926 02:12:10.948770 6854 leader_election.cc:302] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2450ada3a46d4e108e5bf86bd674a231, ed8206169bc84595a5769a74a8a3716d; no voters:
I20260926 02:12:10.949149 6989 raft_consensus.cc:2781] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:10.949641 6989 raft_consensus.cc:686] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 [term 1 LEADER]: Becoming Leader. State: Replica: 2450ada3a46d4e108e5bf86bd674a231, State: Running, Role: LEADER
I20260926 02:12:10.950174 6989 consensus_queue.cc:237] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b87c4100924c70960523a73c3f6257" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33671 } } peers { permanent_uuid: "2450ada3a46d4e108e5bf86bd674a231" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35795 } } peers { permanent_uuid: "ed8206169bc84595a5769a74a8a3716d" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 43589 } }
I20260926 02:12:10.955168 6514 catalog_manager.cc:5495] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2450ada3a46d4e108e5bf86bd674a231 (127.18.159.131). New cstate: current_term: 1 leader_uuid: "2450ada3a46d4e108e5bf86bd674a231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b87c4100924c70960523a73c3f6257" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2450ada3a46d4e108e5bf86bd674a231" member_type: VOTER last_known_addr { host: "127.18.159.131" port: 35795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed8206169bc84595a5769a74a8a3716d" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 43589 } health_report { overall_health: UNKNOWN } } }
I20260926 02:12:11.011518 6979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:11.005744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59616 (local address 127.18.159.131:35795)
0926 02:12:11.005971 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:11.005980 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:11.006000 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:12:11.006292 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:11.006296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:11.006387 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:11.006555 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:11.006946 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:11.007712 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:11.009245 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:11.009660 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:11.009829 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:11.010376 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:11.010906 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:11.011075 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:11.011173 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 02:12:11.011263 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 02:12:11.020066 6788 raft_consensus.cc:1261] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d [term 1 FOLLOWER]: Refusing update from remote peer 2450ada3a46d4e108e5bf86bd674a231: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:12:11.020077 6656 raft_consensus.cc:1261] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257 [term 1 FOLLOWER]: Refusing update from remote peer 2450ada3a46d4e108e5bf86bd674a231: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:12:11.021155 6989 consensus_queue.cc:1035] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79b87c4100924c70960523a73c3f6257" member_type: VOTER last_known_addr { host: "127.18.159.129" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:12:11.021556 6993 consensus_queue.cc:1035] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed8206169bc84595a5769a74a8a3716d" member_type: VOTER last_known_addr { host: "127.18.159.130" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:12:11.052531 6513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:12:11.052860 6513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:12:11.057819 6513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c2bef36a0f54760a7ba776549f73b7e: Sending DeleteTablet for 3 replicas of tablet a705fe2381244c6c80b416d2fdf11a1f
I20260926 02:12:11.059258 6636 tablet_service.cc:1514] Processing DeleteTablet for tablet a705fe2381244c6c80b416d2fdf11a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:12:11 UTC) from {username='slave'} at 127.0.0.1:37432
I20260926 02:12:11.059608 6899 tablet_service.cc:1514] Processing DeleteTablet for tablet a705fe2381244c6c80b416d2fdf11a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:12:11 UTC) from {username='slave'} at 127.0.0.1:59600
I20260926 02:12:11.060397 6768 tablet_service.cc:1514] Processing DeleteTablet for tablet a705fe2381244c6c80b416d2fdf11a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:12:11 UTC) from {username='slave'} at 127.0.0.1:43360
I20260926 02:12:11.060536 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 6573
I20260926 02:12:11.062724 7006 tablet_replica.cc:330] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257: stopping tablet replica
I20260926 02:12:11.062947 7007 tablet_replica.cc:330] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231: stopping tablet replica
I20260926 02:12:11.063436 7006 raft_consensus.cc:2229] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:12:11.063544 7007 raft_consensus.cc:2229] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:12:11.064235 7006 raft_consensus.cc:2258] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:12:11.064628 7007 raft_consensus.cc:2258] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:12:11.067291 7006 ts_tablet_manager.cc:1905] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:12:11.067308 7007 ts_tablet_manager.cc:1905] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:12:11.073187 7008 tablet_replica.cc:330] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d: stopping tablet replica
I20260926 02:12:11.073683 7008 raft_consensus.cc:2229] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:12:11.074244 7008 raft_consensus.cc:2258] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:12:11.074987 7006 ts_tablet_manager.cc:1918] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:12:11.075214 7006 log.cc:1196] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-0/wal/wals/a705fe2381244c6c80b416d2fdf11a1f
I20260926 02:12:11.075810 7006 ts_tablet_manager.cc:1939] T a705fe2381244c6c80b416d2fdf11a1f P 79b87c4100924c70960523a73c3f6257: Deleting consensus metadata
I20260926 02:12:11.076486 7007 ts_tablet_manager.cc:1918] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:12:11.076710 7007 log.cc:1196] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-2/wal/wals/a705fe2381244c6c80b416d2fdf11a1f
I20260926 02:12:11.077275 7007 ts_tablet_manager.cc:1939] T a705fe2381244c6c80b416d2fdf11a1f P 2450ada3a46d4e108e5bf86bd674a231: Deleting consensus metadata
I20260926 02:12:11.078118 6499 catalog_manager.cc:4841] TS 79b87c4100924c70960523a73c3f6257 (127.18.159.129:33671): tablet a705fe2381244c6c80b416d2fdf11a1f (table test-table [id=dab25ef006d34646be8d60a9f9c63bbc]) successfully deleted
I20260926 02:12:11.079265 6501 catalog_manager.cc:4841] TS 2450ada3a46d4e108e5bf86bd674a231 (127.18.159.131:35795): tablet a705fe2381244c6c80b416d2fdf11a1f (table test-table [id=dab25ef006d34646be8d60a9f9c63bbc]) successfully deleted
I20260926 02:12:11.080928 7008 ts_tablet_manager.cc:1905] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:12:11.083102 6977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:11.061640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.18.159.129:33671)
0926 02:12:11.061868 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:11.061878 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:11.072715 (+ 10837us) server_negotiation.cc:407] Connection header received
0926 02:12:11.074708 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:11.074716 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:11.074864 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:11.074977 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:11.076139 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:11.077346 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:11.079482 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:11.079990 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:11.080064 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:11.081562 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:11.081572 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:11.081636 (+ 64us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:11.082246 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:11.082295 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:11.082315 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:11.082456 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:11.082594 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 02:12:11.082709 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 02:12:11.083804 6696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:11.088764 7008 ts_tablet_manager.cc:1918] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:12:11.088974 7008 log.cc:1196] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d: Deleting WAL directory at /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388402166281-19070-0/minicluster-data/ts-1/wal/wals/a705fe2381244c6c80b416d2fdf11a1f
I20260926 02:12:11.089545 7008 ts_tablet_manager.cc:1939] T a705fe2381244c6c80b416d2fdf11a1f P ed8206169bc84595a5769a74a8a3716d: Deleting consensus metadata
I20260926 02:12:11.091657 6501 catalog_manager.cc:4841] TS ed8206169bc84595a5769a74a8a3716d (127.18.159.130:43589): tablet a705fe2381244c6c80b416d2fdf11a1f (table test-table [id=dab25ef006d34646be8d60a9f9c63bbc]) successfully deleted
I20260926 02:12:11.174844 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 6573
I20260926 02:12:11.199621 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 6705
I20260926 02:12:11.207168 6980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:11.200661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43362 (local address 127.18.159.130:43589)
0926 02:12:11.200875 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:11.200881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:11.200902 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:12:11.200978 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:11.200984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:11.201102 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:11.201215 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:11.201615 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:11.202425 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:11.203753 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:11.204310 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:11.204458 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:11.205716 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:11.205725 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:11.205740 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:11.206345 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:11.206403 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:11.206424 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:11.206553 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:11.206698 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 02:12:11.206824 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:12:11.207957 6828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:11.299104 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 6705
I20260926 02:12:11.322691 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 6836
I20260926 02:12:11.330147 6979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:11.323565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59630 (local address 127.18.159.131:35795)
0926 02:12:11.323797 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:11.323803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:11.323823 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:12:11.324019 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:11.324023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:11.324125 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:11.324262 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:11.324674 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:11.325458 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:11.326727 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:11.327209 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:11.327362 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:11.328664 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:11.328673 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:11.328686 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:11.329303 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:11.329366 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:11.329391 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:11.329532 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:11.329680 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 02:12:11.329806 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 02:12:11.330807 6959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:11.450912 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 6836
I20260926 02:12:11.469822 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 6483
I20260926 02:12:11.470842 6544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:11.585700 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 6483
2026-09-26T02:12:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3086 ms)
[----------] 6 tests from EncryptionPolicyTest (22418 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:12:11 dist-test-slave-03rn krb5kdc[7016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:12:11 dist-test-slave-03rn krb5kdc[7016](info): set up 2 sockets
Sep 26 02:12:11 dist-test-slave-03rn krb5kdc[7016](info): commencing operation
krb5kdc: starting...
W20260926 02:12:12.713090 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:12:12 dist-test-slave-03rn krb5kdc[7016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:12:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:12:12.980491 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:13.324724 7029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:13.325117 7029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:13.325309 7029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:13.332993 7029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:13.333184 7029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:13.333305 7029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:13.333421 7029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:13.333509 7029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:13.343750 7029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34977
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:13.345270 7029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:13.346915 7029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:13.359697 7035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:13.359679 7034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:13.359683 7037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:13.360600 7029 server_base.cc:1034] running on GCE node
I20260926 02:12:13.361423 7029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:13.362922 7029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:13.364212 7029 hybrid_clock.cc:648] HybridClock initialized: now 1790388733364125 us; error 66 us; skew 500 ppm
I20260926 02:12:13.364627 7029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:13 dist-test-slave-03rn krb5kdc[7016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:13.373849 7029 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:12:13.375636 7029 webserver.cc:460] Webserver started at http://127.0.0.1:46737/ using document root <none> and password file <none>
I20260926 02:12:13.376536 7029 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:13.376698 7029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:13.377027 7029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:13.380304 7029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "6f530bf833334a5997795e507681fa49"
format_stamp: "Formatted at 2026-09-26 02:12:13 on dist-test-slave-03rn"
server_key: "213d6bb52190dd34093a11ac92ba3384"
server_key_iv: "f282987584229b5d5215d6cf86da2f36"
server_key_version: "encryptionkey@0"
I20260926 02:12:13.381178 7029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "6f530bf833334a5997795e507681fa49"
format_stamp: "Formatted at 2026-09-26 02:12:13 on dist-test-slave-03rn"
server_key: "213d6bb52190dd34093a11ac92ba3384"
server_key_iv: "f282987584229b5d5215d6cf86da2f36"
server_key_version: "encryptionkey@0"
I20260926 02:12:13.386879 7029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:12:13.390909 7044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:13.392585 7029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:12:13.392796 7029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "6f530bf833334a5997795e507681fa49"
format_stamp: "Formatted at 2026-09-26 02:12:13 on dist-test-slave-03rn"
server_key: "213d6bb52190dd34093a11ac92ba3384"
server_key_iv: "f282987584229b5d5215d6cf86da2f36"
server_key_version: "encryptionkey@0"
I20260926 02:12:13.393019 7029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:13.410355 7029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:13.415272 7029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:13.415591 7029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:13.435614 7029 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34977
I20260926 02:12:13.435637 7095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34977 every 8 connection(s)
I20260926 02:12:13.437168 7029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:13.441038 7096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:13.446405 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 7029
I20260926 02:12:13.446614 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:13.447233 19070 external_mini_cluster.cc:1428] Setting key 0b17419f0bbaf71e23103b86b89019ae
I20260926 02:12:13.452802 7096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49: Bootstrap starting.
I20260926 02:12:13.456913 7096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:13.458268 7096 log.cc:824] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:13.461549 7096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49: No bootstrap required, opened a new log
Sep 26 02:12:13 dist-test-slave-03rn krb5kdc[7016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 02:12:13.472666 7096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f530bf833334a5997795e507681fa49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34977 } }
I20260926 02:12:13.473248 7096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:13.473371 7096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f530bf833334a5997795e507681fa49, State: Initialized, Role: FOLLOWER
I20260926 02:12:13.473765 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:13.449267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49187 (local address 127.0.0.1:34977)
0926 02:12:13.450172 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:13.450185 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:13.450229 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:12:13.451997 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:13.452024 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:13.452578 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:13.452937 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:13.454855 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:13.456327 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:13.458282 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:13.459081 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:13.459255 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:13.463577 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:13.463593 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:13.463614 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:13.463670 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:13.468016 (+ 4346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:13.469065 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:13.469078 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:13.469094 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:13.469340 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:13.469872 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:13.469881 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:13.469884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:13.470704 (+ 820us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:13.471044 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:13.471726 (+ 682us) server_negotiation.cc:299] Negotiation successful
0926 02:12:13.472135 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":405,"threads_started":1}
I20260926 02:12:13.474112 7096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f530bf833334a5997795e507681fa49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34977 } }
I20260926 02:12:13.474428 7096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:12:13.474587 7096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:12:13.474773 7096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:13.478044 7096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f530bf833334a5997795e507681fa49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34977 } }
I20260926 02:12:13.478642 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:12:13.478645 7096 leader_election.cc:302] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f530bf833334a5997795e507681fa49; no voters:
I20260926 02:12:13.479352 7096 leader_election.cc:288] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:12:13.479502 7101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:13.481546 7096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:13.487021 7101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 6f530bf833334a5997795e507681fa49, State: Running, Role: LEADER
I20260926 02:12:13.487857 7101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f530bf833334a5997795e507681fa49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34977 } }
I20260926 02:12:13.491302 7103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f530bf833334a5997795e507681fa49. Latest consensus state: current_term: 1 leader_uuid: "6f530bf833334a5997795e507681fa49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f530bf833334a5997795e507681fa49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34977 } } }
I20260926 02:12:13.492255 7103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:13.491824 7102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f530bf833334a5997795e507681fa49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f530bf833334a5997795e507681fa49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34977 } } }
I20260926 02:12:13.492956 7102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49 [sys.catalog]: This master's current role is: LEADER
W20260926 02:12:13.505614 7120 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:12:13.505877 7120 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:12:13.506050 7121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:13.508526 7121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:13.509011 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:13.489180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44123 (local address 127.0.0.1:34977)
0926 02:12:13.489374 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:13.489380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:13.489856 (+ 476us) server_negotiation.cc:407] Connection header received
0926 02:12:13.490050 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:13.490055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:13.490137 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:13.490285 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:13.492734 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:13.494365 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:13.497064 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:13.498160 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:13.498354 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:13.502236 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:13.502241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:13.502245 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:13.502282 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:13.505550 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:13.506630 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:13.506634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:13.506637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:13.506783 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:13.507308 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:13.507312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:13.507314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:13.507683 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:13.507979 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:13.508411 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 02:12:13.508613 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:12:13.519830 7121 catalog_manager.cc:1347] Generated new cluster ID: c21494174e904dbeb9608bcf47d08659
I20260926 02:12:13.520026 7121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:13.547660 7121 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:13.549083 7121 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:13.563364 7121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f530bf833334a5997795e507681fa49: Generated new TSK 0
I20260926 02:12:13.564566 7121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:12:13.574434 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 7029
I20260926 02:12:13.575870 7091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:13.672284 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 7029
W20260926 02:12:13.689682 7014 connection.cc:535] client connection to 127.0.0.1:34977 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34977): Connection reset by peer (error 104)
2026-09-26T02:12:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2078 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:12:13 dist-test-slave-03rn krb5kdc[7128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:12:13 dist-test-slave-03rn krb5kdc[7128](info): set up 2 sockets
Sep 26 02:12:13 dist-test-slave-03rn krb5kdc[7128](info): commencing operation
krb5kdc: starting...
W20260926 02:12:14.780876 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:12:14 dist-test-slave-03rn krb5kdc[7128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:12:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:12:15.048322 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44821
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:15.392841 7141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:15.393169 7141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:15.393360 7141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:15.401234 7141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:15.401364 7141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:15.401489 7141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:15.401566 7141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:15.401605 7141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:15.412194 7141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44821
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:15.413604 7141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:15.414959 7141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:15.427903 7146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:15.427980 7147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:15.427938 7149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:15.428879 7141 server_base.cc:1034] running on GCE node
I20260926 02:12:15.429734 7141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:15.431277 7141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:15.432554 7141 hybrid_clock.cc:648] HybridClock initialized: now 1790388735432485 us; error 55 us; skew 500 ppm
I20260926 02:12:15.432983 7141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:15 dist-test-slave-03rn krb5kdc[7128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:15.436537 7141 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:12:15.438294 7141 webserver.cc:460] Webserver started at http://127.0.0.1:36581/ using document root <none> and password file <none>
I20260926 02:12:15.439037 7141 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:15.439218 7141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:15.439553 7141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:15.442835 7141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "4b0e6f1e4b744c50895fab7471bc43d9"
format_stamp: "Formatted at 2026-09-26 02:12:15 on dist-test-slave-03rn"
server_key: "24622475d1e6c1e6016ce847e49b3df5"
server_key_iv: "00498279a8a6ec2eca3e010026e8936a"
server_key_version: "encryptionkey@0"
I20260926 02:12:15.443696 7141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "4b0e6f1e4b744c50895fab7471bc43d9"
format_stamp: "Formatted at 2026-09-26 02:12:15 on dist-test-slave-03rn"
server_key: "24622475d1e6c1e6016ce847e49b3df5"
server_key_iv: "00498279a8a6ec2eca3e010026e8936a"
server_key_version: "encryptionkey@0"
I20260926 02:12:15.449533 7141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:12:15.453498 7156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:15.455027 7141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:12:15.455250 7141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "4b0e6f1e4b744c50895fab7471bc43d9"
format_stamp: "Formatted at 2026-09-26 02:12:15 on dist-test-slave-03rn"
server_key: "24622475d1e6c1e6016ce847e49b3df5"
server_key_iv: "00498279a8a6ec2eca3e010026e8936a"
server_key_version: "encryptionkey@0"
I20260926 02:12:15.455485 7141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:15.493116 7141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:15.497684 7141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:15.498006 7141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:15.517611 7141 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44821
I20260926 02:12:15.517632 7207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44821 every 8 connection(s)
I20260926 02:12:15.519114 7141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:15.523095 7208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:15.525301 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 7141
I20260926 02:12:15.525511 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:15.525921 19070 external_mini_cluster.cc:1428] Setting key 0e480e5ffbccebcc2b46c26dceb117df
I20260926 02:12:15.534946 7208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9: Bootstrap starting.
I20260926 02:12:15.538194 7208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:15.539417 7208 log.cc:824] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:15.539450 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:15.527757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32965 (local address 127.0.0.1:44821)
0926 02:12:15.528645 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:15.528657 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:15.528703 (+ 46us) server_negotiation.cc:407] Connection header received
0926 02:12:15.530491 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:15.530515 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:15.530945 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:15.531348 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:15.531923 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:15.533391 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:15.534778 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:15.535570 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:15.535767 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:15.537100 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:15.537108 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:15.537123 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:15.537681 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:15.537748 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:15.537776 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:15.537945 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:15.538244 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 02:12:15.538445 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":400,"threads_started":1}
I20260926 02:12:15.542583 7208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9: No bootstrap required, opened a new log
I20260926 02:12:15.543635 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:12:15.551844 7208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0e6f1e4b744c50895fab7471bc43d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44821 } }
I20260926 02:12:15.552454 7208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:15.552580 7208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0e6f1e4b744c50895fab7471bc43d9, State: Initialized, Role: FOLLOWER
I20260926 02:12:15.553205 7208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0e6f1e4b744c50895fab7471bc43d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44821 } }
I20260926 02:12:15.553519 7208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:12:15.553668 7208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:12:15.553872 7208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:15.556330 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:15.548211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48989 (local address 127.0.0.1:44821)
0926 02:12:15.548451 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:15.548461 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:15.548742 (+ 281us) server_negotiation.cc:407] Connection header received
0926 02:12:15.549018 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:15.549052 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:15.549188 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:15.549323 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:15.550034 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:15.551216 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:15.553341 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:15.554097 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:15.554267 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:15.555380 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:15.555387 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:15.555392 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:15.555416 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:15.555465 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:15.555491 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:15.555615 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:15.555912 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 02:12:15.555969 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 02:12:15.556900 7208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0e6f1e4b744c50895fab7471bc43d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44821 } }
I20260926 02:12:15.557461 7208 leader_election.cc:302] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b0e6f1e4b744c50895fab7471bc43d9; no voters:
I20260926 02:12:15.558162 7208 leader_election.cc:288] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:12:15.558342 7218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:15.559115 7218 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9 [term 1 LEADER]: Becoming Leader. State: Replica: 4b0e6f1e4b744c50895fab7471bc43d9, State: Running, Role: LEADER
I20260926 02:12:15.559791 7218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0e6f1e4b744c50895fab7471bc43d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44821 } }
I20260926 02:12:15.560504 7208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:15.564276 7220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b0e6f1e4b744c50895fab7471bc43d9. Latest consensus state: current_term: 1 leader_uuid: "4b0e6f1e4b744c50895fab7471bc43d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0e6f1e4b744c50895fab7471bc43d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44821 } } }
I20260926 02:12:15.564815 7220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:15.565582 7219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b0e6f1e4b744c50895fab7471bc43d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0e6f1e4b744c50895fab7471bc43d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44821 } } }
I20260926 02:12:15.565897 7230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:15.565963 7219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:15.572901 7230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:15.590709 7230 catalog_manager.cc:1347] Generated new cluster ID: 01d9cbc0c5044122a7a47d592a48dc16
I20260926 02:12:15.590852 7230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:15.639043 7230 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:15.640574 7230 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:15.648233 7230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b0e6f1e4b744c50895fab7471bc43d9: Generated new TSK 0
I20260926 02:12:15.649127 7230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:12:15.657267 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 7141
W20260926 02:12:15.658469 7203 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:32965
W20260926 02:12:15.659332 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 7141: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:12:15.659523 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 7141
W20260926 02:12:15.676946 7126 connection.cc:535] client connection to 127.0.0.1:44821 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44821): Connection reset by peer (error 104)
2026-09-26T02:12:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1987 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:12:15 dist-test-slave-03rn krb5kdc[7240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:12:15 dist-test-slave-03rn krb5kdc[7240](info): set up 2 sockets
Sep 26 02:12:15 dist-test-slave-03rn krb5kdc[7240](info): commencing operation
krb5kdc: starting...
W20260926 02:12:16.789069 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:12:16 dist-test-slave-03rn krb5kdc[7240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:12:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:12:17.060647 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:17.426188 7253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:17.426546 7253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:17.426739 7253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:17.434338 7253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:17.434473 7253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:17.434578 7253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:17.434692 7253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:17.434772 7253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:17.445170 7253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32947
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:17.446589 7253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:17.447960 7253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:17.460561 7258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:17.460634 7261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:17.460650 7259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:17.461606 7253 server_base.cc:1034] running on GCE node
I20260926 02:12:17.462455 7253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:17.464017 7253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:17.465263 7253 hybrid_clock.cc:648] HybridClock initialized: now 1790388737465199 us; error 74 us; skew 500 ppm
I20260926 02:12:17.465683 7253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:17 dist-test-slave-03rn krb5kdc[7240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:17.469112 7253 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:12:17.470640 7253 webserver.cc:460] Webserver started at http://127.0.0.1:36369/ using document root <none> and password file <none>
I20260926 02:12:17.471328 7253 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:17.471436 7253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:17.471697 7253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:17.474887 7253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "a132eb1eb8b24502a8b19f627958bc24"
format_stamp: "Formatted at 2026-09-26 02:12:17 on dist-test-slave-03rn"
server_key: "71ec1b12646ef0c465cacad3c612b02f"
server_key_iv: "3be792451b46da0dfdec4fef2401508a"
server_key_version: "encryptionkey@0"
I20260926 02:12:17.475679 7253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "a132eb1eb8b24502a8b19f627958bc24"
format_stamp: "Formatted at 2026-09-26 02:12:17 on dist-test-slave-03rn"
server_key: "71ec1b12646ef0c465cacad3c612b02f"
server_key_iv: "3be792451b46da0dfdec4fef2401508a"
server_key_version: "encryptionkey@0"
I20260926 02:12:17.481388 7253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:12:17.485148 7268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:17.486655 7253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:12:17.486876 7253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "a132eb1eb8b24502a8b19f627958bc24"
format_stamp: "Formatted at 2026-09-26 02:12:17 on dist-test-slave-03rn"
server_key: "71ec1b12646ef0c465cacad3c612b02f"
server_key_iv: "3be792451b46da0dfdec4fef2401508a"
server_key_version: "encryptionkey@0"
I20260926 02:12:17.487103 7253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:17.493033 7253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:17.497632 7253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:17.497957 7253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:17.517110 7253 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32947
I20260926 02:12:17.517160 7319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32947 every 8 connection(s)
I20260926 02:12:17.518707 7253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:17.522614 7320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:17.528580 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 7253
I20260926 02:12:17.528786 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:17.529208 19070 external_mini_cluster.cc:1428] Setting key 5bc631384e44daee4fe0e0f9ec389a05
I20260926 02:12:17.534158 7320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24: Bootstrap starting.
I20260926 02:12:17.537465 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:17.531044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35045 (local address 127.0.0.1:32947)
0926 02:12:17.531971 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:17.531984 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:17.532028 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:12:17.533827 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:17.533850 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:17.534210 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:17.534682 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:17.535747 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:17.535755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:17.535767 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:17.535784 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:17.535841 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:17.535861 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:17.536102 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:17.536290 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 02:12:17.536467 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":410,"threads_started":1}
I20260926 02:12:17.537808 7320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:17.539134 7320 log.cc:824] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:17.541854 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:12:17.542445 7320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24: No bootstrap required, opened a new log
I20260926 02:12:17.549291 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:17.546444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54315 (local address 127.0.0.1:32947)
0926 02:12:17.546619 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:17.546624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:17.547092 (+ 468us) server_negotiation.cc:407] Connection header received
0926 02:12:17.547240 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:17.547245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:17.547327 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:17.547439 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:17.548570 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:17.548577 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:17.548582 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:17.548603 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:17.548660 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:17.548681 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:17.548824 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:17.548913 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 02:12:17.548982 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:12:17.551991 7320 raft_consensus.cc:348] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a132eb1eb8b24502a8b19f627958bc24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32947 } }
I20260926 02:12:17.552505 7320 raft_consensus.cc:374] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:17.552635 7320 raft_consensus.cc:729] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a132eb1eb8b24502a8b19f627958bc24, State: Initialized, Role: FOLLOWER
I20260926 02:12:17.553232 7320 consensus_queue.cc:260] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a132eb1eb8b24502a8b19f627958bc24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32947 } }
I20260926 02:12:17.553498 7320 raft_consensus.cc:388] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:12:17.553658 7320 raft_consensus.cc:482] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:12:17.553848 7320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:17.556514 7320 raft_consensus.cc:504] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a132eb1eb8b24502a8b19f627958bc24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32947 } }
I20260926 02:12:17.557099 7320 leader_election.cc:302] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a132eb1eb8b24502a8b19f627958bc24; no voters:
I20260926 02:12:17.557858 7320 leader_election.cc:288] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:12:17.558017 7330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:17.560662 7320 sys_catalog.cc:564] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:17.561061 7330 raft_consensus.cc:686] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24 [term 1 LEADER]: Becoming Leader. State: Replica: a132eb1eb8b24502a8b19f627958bc24, State: Running, Role: LEADER
I20260926 02:12:17.561654 7330 consensus_queue.cc:237] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a132eb1eb8b24502a8b19f627958bc24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32947 } }
I20260926 02:12:17.565258 7332 sys_catalog.cc:455] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a132eb1eb8b24502a8b19f627958bc24. Latest consensus state: current_term: 1 leader_uuid: "a132eb1eb8b24502a8b19f627958bc24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a132eb1eb8b24502a8b19f627958bc24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32947 } } }
I20260926 02:12:17.565766 7332 sys_catalog.cc:458] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:17.566074 7331 sys_catalog.cc:455] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a132eb1eb8b24502a8b19f627958bc24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a132eb1eb8b24502a8b19f627958bc24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32947 } } }
I20260926 02:12:17.566416 7331 sys_catalog.cc:458] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:17.567380 7343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:17.572893 7343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:17.582810 7343 catalog_manager.cc:1347] Generated new cluster ID: 9b0fc45c6fc04197a479493f809f2a95
I20260926 02:12:17.582952 7343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:17.593710 7343 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:17.594182 7343 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:17.613649 7343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a132eb1eb8b24502a8b19f627958bc24: Generated new TSK 0
I20260926 02:12:17.614714 7343 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:12:17.622607 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 7253
W20260926 02:12:17.623837 7315 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35045
W20260926 02:12:17.624871 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 7253: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:12:17.625072 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 7253
2026-09-26T02:12:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1964 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:12:17 dist-test-slave-03rn krb5kdc[7351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:12:17 dist-test-slave-03rn krb5kdc[7351](info): set up 2 sockets
Sep 26 02:12:17 dist-test-slave-03rn krb5kdc[7351](info): commencing operation
krb5kdc: starting...
W20260926 02:12:18.781126 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:12:18 dist-test-slave-03rn krb5kdc[7351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:12:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:12:19.056479 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44231
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:19.404699 7364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:19.405014 7364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:19.405222 7364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:19.412778 7364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:19.412902 7364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:19.412994 7364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:19.413097 7364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:19.413180 7364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:19.423532 7364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:19.424928 7364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:19.426462 7364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:19.439061 7372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:19.439033 7369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:19.439118 7370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:19.440019 7364 server_base.cc:1034] running on GCE node
I20260926 02:12:19.440893 7364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:19.442484 7364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:19.443763 7364 hybrid_clock.cc:648] HybridClock initialized: now 1790388739443729 us; error 79 us; skew 500 ppm
I20260926 02:12:19.444221 7364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:19 dist-test-slave-03rn krb5kdc[7351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:19.453003 7364 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:12:19.454736 7364 webserver.cc:460] Webserver started at http://127.0.0.1:37785/ using document root <none> and password file <none>
I20260926 02:12:19.455426 7364 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:19.455530 7364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:19.455793 7364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:19.458926 7364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "d3cf13c1a905484a8c8d57ba3340e862"
format_stamp: "Formatted at 2026-09-26 02:12:19 on dist-test-slave-03rn"
server_key: "5230f61fa36ca58bf384413a9d89c8d3"
server_key_iv: "1aad086ac84565c0efbe8a6bce2b19c2"
server_key_version: "encryptionkey@0"
I20260926 02:12:19.459723 7364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "d3cf13c1a905484a8c8d57ba3340e862"
format_stamp: "Formatted at 2026-09-26 02:12:19 on dist-test-slave-03rn"
server_key: "5230f61fa36ca58bf384413a9d89c8d3"
server_key_iv: "1aad086ac84565c0efbe8a6bce2b19c2"
server_key_version: "encryptionkey@0"
I20260926 02:12:19.465369 7364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:12:19.469427 7379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:19.471127 7364 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:12:19.471369 7364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "d3cf13c1a905484a8c8d57ba3340e862"
format_stamp: "Formatted at 2026-09-26 02:12:19 on dist-test-slave-03rn"
server_key: "5230f61fa36ca58bf384413a9d89c8d3"
server_key_iv: "1aad086ac84565c0efbe8a6bce2b19c2"
server_key_version: "encryptionkey@0"
I20260926 02:12:19.471606 7364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:19.491140 7364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:19.495812 7364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:19.496131 7364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:19.516106 7364 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44231
I20260926 02:12:19.516129 7430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44231 every 8 connection(s)
I20260926 02:12:19.517679 7364 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:19.521689 7431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:19.523861 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 7364
I20260926 02:12:19.524055 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:19.524647 19070 external_mini_cluster.cc:1428] Setting key 781adc3589468fa1d9ae6b10b7a3e2f9
I20260926 02:12:19.537590 7431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862: Bootstrap starting.
Sep 26 02:12:19 dist-test-slave-03rn krb5kdc[7351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 02:12:19.540979 7431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:19.542299 7431 log.cc:824] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:19.545554 7431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862: No bootstrap required, opened a new log
I20260926 02:12:19.549371 7434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:19.526595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33387 (local address 127.0.0.1:44231)
0926 02:12:19.527574 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:19.527587 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:19.527631 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:12:19.529548 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:19.529575 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:19.530061 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:19.530516 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:19.532551 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:19.534177 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:19.535736 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:19.536620 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:19.536848 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:19.541222 (+ 4374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:19.541232 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:19.541249 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:19.541299 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:19.544738 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:19.545557 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:19.545566 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:19.545573 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:19.545808 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:19.546233 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:19.546239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:19.546242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:19.546847 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:19.547192 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:19.547690 (+ 498us) server_negotiation.cc:299] Negotiation successful
0926 02:12:19.548269 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":526,"threads_started":1}
I20260926 02:12:19.553508 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:12:19.554373 7431 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3cf13c1a905484a8c8d57ba3340e862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44231 } }
I20260926 02:12:19.554829 7431 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:19.554949 7431 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cf13c1a905484a8c8d57ba3340e862, State: Initialized, Role: FOLLOWER
I20260926 02:12:19.555549 7431 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3cf13c1a905484a8c8d57ba3340e862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44231 } }
I20260926 02:12:19.555814 7431 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:12:19.555959 7431 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:12:19.556133 7431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:19.559232 7431 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3cf13c1a905484a8c8d57ba3340e862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44231 } }
I20260926 02:12:19.559808 7431 leader_election.cc:302] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3cf13c1a905484a8c8d57ba3340e862; no voters:
I20260926 02:12:19.560555 7431 leader_election.cc:288] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:12:19.560672 7441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:19.563449 7431 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:19.564474 7441 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862 [term 1 LEADER]: Becoming Leader. State: Replica: d3cf13c1a905484a8c8d57ba3340e862, State: Running, Role: LEADER
I20260926 02:12:19.565802 7441 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3cf13c1a905484a8c8d57ba3340e862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44231 } }
I20260926 02:12:19.571368 7443 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3cf13c1a905484a8c8d57ba3340e862. Latest consensus state: current_term: 1 leader_uuid: "d3cf13c1a905484a8c8d57ba3340e862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3cf13c1a905484a8c8d57ba3340e862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44231 } } }
I20260926 02:12:19.572877 7443 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:19.572196 7442 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3cf13c1a905484a8c8d57ba3340e862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3cf13c1a905484a8c8d57ba3340e862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44231 } } }
I20260926 02:12:19.573850 7442 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:19.574487 7447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:19.580222 7434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:19.558382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56273 (local address 127.0.0.1:44231)
0926 02:12:19.558595 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:19.558603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:19.558981 (+ 378us) server_negotiation.cc:407] Connection header received
0926 02:12:19.559207 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:19.559213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:19.559301 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:19.559410 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:19.561481 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:19.562375 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:19.563793 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:19.564496 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:19.564641 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:19.570668 (+ 6027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:19.570677 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:19.570681 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:19.570731 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:19.575339 (+ 4608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:19.576814 (+ 1475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:19.576824 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:19.576830 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:19.577035 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:19.577652 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:19.577660 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:19.577667 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:19.578207 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:19.578528 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:19.579345 (+ 817us) server_negotiation.cc:299] Negotiation successful
0926 02:12:19.579705 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:12:19.588370 7447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:19.600288 7447 catalog_manager.cc:1347] Generated new cluster ID: e0cfb638f46a4745bfb75b38a719f867
I20260926 02:12:19.600430 7447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:19.633634 7447 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:19.635548 7447 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:19.646379 7447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3cf13c1a905484a8c8d57ba3340e862: Generated new TSK 0
I20260926 02:12:19.647624 7447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:12:19.662771 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 7364
I20260926 02:12:19.664045 7426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:19.760814 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 7364
2026-09-26T02:12:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2136 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:12:19 dist-test-slave-03rn krb5kdc[7463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:12:19 dist-test-slave-03rn krb5kdc[7463](info): set up 2 sockets
Sep 26 02:12:19 dist-test-slave-03rn krb5kdc[7463](info): commencing operation
krb5kdc: starting...
W20260926 02:12:20.869084 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:12:21 dist-test-slave-03rn krb5kdc[7463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:12:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:12:21.141642 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:21.485788 7476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:21.486104 7476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:21.486281 7476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:21.493721 7476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:21.493855 7476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:21.493958 7476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:21.494052 7476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:21.494163 7476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:21.504383 7476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:21.505757 7476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:21.507174 7476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:21.519876 7481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:21.519901 7484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:21.519842 7482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:21.521231 7476 server_base.cc:1034] running on GCE node
I20260926 02:12:21.522099 7476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:21.523659 7476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:21.524915 7476 hybrid_clock.cc:648] HybridClock initialized: now 1790388741524867 us; error 46 us; skew 500 ppm
I20260926 02:12:21.525298 7476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:21 dist-test-slave-03rn krb5kdc[7463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:21.528622 7476 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:12:21.530254 7476 webserver.cc:460] Webserver started at http://127.0.0.1:44977/ using document root <none> and password file <none>
I20260926 02:12:21.530978 7476 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:21.531129 7476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:21.531451 7476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:21.534806 7476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "42cadf0d9e3643d4b3f089abbd2ac96a"
format_stamp: "Formatted at 2026-09-26 02:12:21 on dist-test-slave-03rn"
server_key: "a2e3fe3edb5926000c0b33c2c8758148"
server_key_iv: "8cac855c41c775e652c36e40c50c475d"
server_key_version: "encryptionkey@0"
I20260926 02:12:21.535626 7476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "42cadf0d9e3643d4b3f089abbd2ac96a"
format_stamp: "Formatted at 2026-09-26 02:12:21 on dist-test-slave-03rn"
server_key: "a2e3fe3edb5926000c0b33c2c8758148"
server_key_iv: "8cac855c41c775e652c36e40c50c475d"
server_key_version: "encryptionkey@0"
I20260926 02:12:21.542315 7476 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260926 02:12:21.547773 7491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:21.549678 7476 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 02:12:21.549903 7476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "42cadf0d9e3643d4b3f089abbd2ac96a"
format_stamp: "Formatted at 2026-09-26 02:12:21 on dist-test-slave-03rn"
server_key: "a2e3fe3edb5926000c0b33c2c8758148"
server_key_iv: "8cac855c41c775e652c36e40c50c475d"
server_key_version: "encryptionkey@0"
I20260926 02:12:21.550122 7476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:21.582576 7476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:21.587044 7476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:21.587353 7476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:21.607421 7476 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33941
I20260926 02:12:21.607446 7542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33941 every 8 connection(s)
I20260926 02:12:21.608897 7476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:21.612670 7543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:21.618947 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 7476
I20260926 02:12:21.619140 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:21.619704 19070 external_mini_cluster.cc:1428] Setting key 88c9d414f1730c2a262119e8e25fab62
I20260926 02:12:21.624413 7543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a: Bootstrap starting.
I20260926 02:12:21.628139 7543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:21.629494 7543 log.cc:824] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:21.632931 7543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a: No bootstrap required, opened a new log
I20260926 02:12:21.634667 7546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:21.621673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39195 (local address 127.0.0.1:33941)
0926 02:12:21.622556 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:21.622569 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:21.622611 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:12:21.624485 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:21.624511 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:21.625003 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:21.625345 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:21.625996 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:21.627302 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:21.629217 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:21.630297 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:21.630515 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:21.632025 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:21.632033 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:21.632049 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:21.632718 (+ 669us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:21.632806 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:21.632833 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:21.632968 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:21.633240 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 02:12:21.633529 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":439,"threads_started":1}
I20260926 02:12:21.638990 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:12:21.642203 7543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42cadf0d9e3643d4b3f089abbd2ac96a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } }
I20260926 02:12:21.642656 7543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:21.642778 7543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cadf0d9e3643d4b3f089abbd2ac96a, State: Initialized, Role: FOLLOWER
I20260926 02:12:21.643507 7543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42cadf0d9e3643d4b3f089abbd2ac96a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } }
I20260926 02:12:21.643853 7543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:12:21.644017 7543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:12:21.644243 7543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:21.647408 7543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42cadf0d9e3643d4b3f089abbd2ac96a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } }
I20260926 02:12:21.648000 7543 leader_election.cc:302] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42cadf0d9e3643d4b3f089abbd2ac96a; no voters:
I20260926 02:12:21.648782 7543 leader_election.cc:288] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:12:21.648903 7553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:21.650081 7553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a [term 1 LEADER]: Becoming Leader. State: Replica: 42cadf0d9e3643d4b3f089abbd2ac96a, State: Running, Role: LEADER
I20260926 02:12:21.650748 7553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42cadf0d9e3643d4b3f089abbd2ac96a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } }
I20260926 02:12:21.651528 7546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:21.643590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59625 (local address 127.0.0.1:33941)
0926 02:12:21.643862 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:21.643872 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:21.644446 (+ 574us) server_negotiation.cc:407] Connection header received
0926 02:12:21.644690 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:21.644698 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:21.644790 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:21.644928 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:21.645452 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:21.646568 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:21.648105 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:21.648917 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:21.649052 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:21.650448 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:21.650454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:21.650459 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:21.650481 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:21.650533 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:21.650554 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:21.650649 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:21.650864 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 02:12:21.651009 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:12:21.656615 7553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42cadf0d9e3643d4b3f089abbd2ac96a. Latest consensus state: current_term: 1 leader_uuid: "42cadf0d9e3643d4b3f089abbd2ac96a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42cadf0d9e3643d4b3f089abbd2ac96a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } } }
I20260926 02:12:21.656945 7553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:21.658103 7555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42cadf0d9e3643d4b3f089abbd2ac96a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42cadf0d9e3643d4b3f089abbd2ac96a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } } }
I20260926 02:12:21.658459 7543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:21.658555 7555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:21.659093 7557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:21.666158 7557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:21.675824 7557 catalog_manager.cc:1347] Generated new cluster ID: d0575314c75f42fa9ba092ef616d414f
I20260926 02:12:21.675973 7557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:21.692270 7557 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:21.693827 7557 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:21.702118 7557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42cadf0d9e3643d4b3f089abbd2ac96a: Generated new TSK 0
I20260926 02:12:21.703063 7557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:12:21.709625 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 7476
W20260926 02:12:21.711082 7538 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39195
W20260926 02:12:21.711853 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 7476: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:12:21.712011 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 7476
2026-09-26T02:12:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1950 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:12:21 dist-test-slave-03rn krb5kdc[7574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:12:21 dist-test-slave-03rn krb5kdc[7574](info): set up 2 sockets
Sep 26 02:12:21 dist-test-slave-03rn krb5kdc[7574](info): commencing operation
krb5kdc: starting...
W20260926 02:12:22.820899 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:12:22 dist-test-slave-03rn krb5kdc[7574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:12:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:12:23.100328 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41209
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:23.461098 7587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:23.461549 7587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:23.461745 7587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:23.470031 7587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:23.470237 7587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:23.470366 7587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:23.470472 7587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:23.470561 7587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:23.481040 7587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:23.482518 7587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:23.483949 7587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:23.496882 7592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:23.496984 7593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:23.497014 7595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:23.498391 7587 server_base.cc:1034] running on GCE node
I20260926 02:12:23.499402 7587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:23.501149 7587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:23.502421 7587 hybrid_clock.cc:648] HybridClock initialized: now 1790388743502366 us; error 54 us; skew 500 ppm
I20260926 02:12:23.502853 7587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:23 dist-test-slave-03rn krb5kdc[7574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:23.506494 7587 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:12:23.508260 7587 webserver.cc:460] Webserver started at http://127.0.0.1:36799/ using document root <none> and password file <none>
I20260926 02:12:23.508972 7587 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:23.509079 7587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:23.509358 7587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:23.512631 7587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "eb9cdc7d685a43559d8967e9b6668a35"
format_stamp: "Formatted at 2026-09-26 02:12:23 on dist-test-slave-03rn"
server_key: "4fc6f0e7151484b612104f50fe3efd52"
server_key_iv: "40378493c1d6bc9d8a37baccba0194e5"
server_key_version: "encryptionkey@0"
I20260926 02:12:23.513434 7587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "eb9cdc7d685a43559d8967e9b6668a35"
format_stamp: "Formatted at 2026-09-26 02:12:23 on dist-test-slave-03rn"
server_key: "4fc6f0e7151484b612104f50fe3efd52"
server_key_iv: "40378493c1d6bc9d8a37baccba0194e5"
server_key_version: "encryptionkey@0"
I20260926 02:12:23.519176 7587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:12:23.523209 7602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:23.524955 7587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:12:23.525188 7587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "eb9cdc7d685a43559d8967e9b6668a35"
format_stamp: "Formatted at 2026-09-26 02:12:23 on dist-test-slave-03rn"
server_key: "4fc6f0e7151484b612104f50fe3efd52"
server_key_iv: "40378493c1d6bc9d8a37baccba0194e5"
server_key_version: "encryptionkey@0"
I20260926 02:12:23.525401 7587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:23.531682 7587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:23.536654 7587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:23.536989 7587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:23.557798 7587 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41209
I20260926 02:12:23.557838 7653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41209 every 8 connection(s)
I20260926 02:12:23.559422 7587 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:23.563326 7654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:23.567378 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 7587
I20260926 02:12:23.567529 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:23.568002 19070 external_mini_cluster.cc:1428] Setting key 65ecdacd3f3eae9c383a657ad414d778
I20260926 02:12:23.575377 7654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35: Bootstrap starting.
I20260926 02:12:23.577445 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:23.570154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35817 (local address 127.0.0.1:41209)
0926 02:12:23.571201 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:23.571213 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:23.571285 (+ 72us) server_negotiation.cc:407] Connection header received
0926 02:12:23.573671 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:23.573700 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:23.574103 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:23.574492 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:23.575692 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:23.575700 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:23.575718 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:23.575751 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:23.575823 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:23.575847 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:23.576045 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:23.576215 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 02:12:23.576443 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":431,"threads_started":1}
I20260926 02:12:23.579319 7654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:23.580632 7654 log.cc:824] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:23.581959 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:12:23.584264 7654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35: No bootstrap required, opened a new log
I20260926 02:12:23.590642 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:23.587432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52115 (local address 127.0.0.1:41209)
0926 02:12:23.587667 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:23.587672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:23.587993 (+ 321us) server_negotiation.cc:407] Connection header received
0926 02:12:23.588223 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:23.588227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:23.588282 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:23.588383 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:23.589515 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:23.589521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:23.589528 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:23.589547 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:23.589595 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:23.589616 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:23.589703 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:23.590063 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 02:12:23.590174 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 02:12:23.594930 7654 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9cdc7d685a43559d8967e9b6668a35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41209 } }
I20260926 02:12:23.595377 7654 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:23.595505 7654 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9cdc7d685a43559d8967e9b6668a35, State: Initialized, Role: FOLLOWER
I20260926 02:12:23.596109 7654 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9cdc7d685a43559d8967e9b6668a35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41209 } }
I20260926 02:12:23.596448 7654 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:12:23.596601 7654 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:12:23.596791 7654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:23.599213 7654 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9cdc7d685a43559d8967e9b6668a35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41209 } }
I20260926 02:12:23.599783 7654 leader_election.cc:302] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb9cdc7d685a43559d8967e9b6668a35; no voters:
I20260926 02:12:23.600484 7654 leader_election.cc:288] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:12:23.600656 7664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:23.601358 7664 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35 [term 1 LEADER]: Becoming Leader. State: Replica: eb9cdc7d685a43559d8967e9b6668a35, State: Running, Role: LEADER
I20260926 02:12:23.601953 7664 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9cdc7d685a43559d8967e9b6668a35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41209 } }
I20260926 02:12:23.602712 7654 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:23.605026 7666 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb9cdc7d685a43559d8967e9b6668a35. Latest consensus state: current_term: 1 leader_uuid: "eb9cdc7d685a43559d8967e9b6668a35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9cdc7d685a43559d8967e9b6668a35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41209 } } }
I20260926 02:12:23.605531 7666 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:23.606601 7670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:23.608143 7665 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb9cdc7d685a43559d8967e9b6668a35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9cdc7d685a43559d8967e9b6668a35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41209 } } }
I20260926 02:12:23.608525 7665 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:23.617120 7670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:23.628720 7670 catalog_manager.cc:1347] Generated new cluster ID: 7fa93e2bcb0f4c45adacd5c75d0ce199
I20260926 02:12:23.628863 7670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:23.657480 7670 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:23.658056 7670 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:23.683265 7670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb9cdc7d685a43559d8967e9b6668a35: Generated new TSK 0
I20260926 02:12:23.684346 7670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:12:23.696269 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 7587
W20260926 02:12:23.697551 7649 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35817
W20260926 02:12:23.698447 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 7587: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:12:23.698597 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 7587
2026-09-26T02:12:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1986 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:12:23 dist-test-slave-03rn krb5kdc[7685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:12:23 dist-test-slave-03rn krb5kdc[7685](info): set up 2 sockets
Sep 26 02:12:23 dist-test-slave-03rn krb5kdc[7685](info): commencing operation
krb5kdc: starting...
W20260926 02:12:24.805029 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:12:24 dist-test-slave-03rn krb5kdc[7685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:12:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:12:25.076809 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37499
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:25.423986 7698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:25.424327 7698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:25.424507 7698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:25.431986 7698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:25.432111 7698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:25.432253 7698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:25.432382 7698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:25.432479 7698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:25.443120 7698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:25.444677 7698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:25.446067 7698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:25.459249 7703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:25.459286 7704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:25.459336 7706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:25.460266 7698 server_base.cc:1034] running on GCE node
I20260926 02:12:25.461181 7698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:25.462946 7698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:25.464228 7698 hybrid_clock.cc:648] HybridClock initialized: now 1790388745464110 us; error 63 us; skew 500 ppm
I20260926 02:12:25.464588 7698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:25 dist-test-slave-03rn krb5kdc[7685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:25.471995 7698 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:12:25.473701 7698 webserver.cc:460] Webserver started at http://127.0.0.1:42843/ using document root <none> and password file <none>
I20260926 02:12:25.474393 7698 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:25.474499 7698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:25.474761 7698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:25.478363 7698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "9d2218aa81bb41cbbb2579efad7c1a9b"
format_stamp: "Formatted at 2026-09-26 02:12:25 on dist-test-slave-03rn"
server_key: "1d6058e436eb0f9d68ce238af5830d29"
server_key_iv: "62209c0773a3006c1e9d701c5f0b45c5"
server_key_version: "encryptionkey@0"
I20260926 02:12:25.479136 7698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "9d2218aa81bb41cbbb2579efad7c1a9b"
format_stamp: "Formatted at 2026-09-26 02:12:25 on dist-test-slave-03rn"
server_key: "1d6058e436eb0f9d68ce238af5830d29"
server_key_iv: "62209c0773a3006c1e9d701c5f0b45c5"
server_key_version: "encryptionkey@0"
I20260926 02:12:25.484763 7698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 02:12:25.488711 7713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:25.490363 7698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:12:25.490535 7698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "9d2218aa81bb41cbbb2579efad7c1a9b"
format_stamp: "Formatted at 2026-09-26 02:12:25 on dist-test-slave-03rn"
server_key: "1d6058e436eb0f9d68ce238af5830d29"
server_key_iv: "62209c0773a3006c1e9d701c5f0b45c5"
server_key_version: "encryptionkey@0"
I20260926 02:12:25.490726 7698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:25.503150 7698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:25.507638 7698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:25.507901 7698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:25.528062 7698 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37499
I20260926 02:12:25.528100 7764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37499 every 8 connection(s)
I20260926 02:12:25.529583 7698 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:25.533567 7765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:25.533845 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 7698
I20260926 02:12:25.534030 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:25.534495 19070 external_mini_cluster.cc:1428] Setting key 374a72ce1cc125b742e409a0dfa92703
I20260926 02:12:25.549556 7765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b: Bootstrap starting.
Sep 26 02:12:25 dist-test-slave-03rn krb5kdc[7685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 02:12:25.554128 7765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:25.555697 7765 log.cc:824] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:25.558897 7765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b: No bootstrap required, opened a new log
I20260926 02:12:25.561256 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:25.536540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:49321 (local address 127.0.0.1:37499)
0926 02:12:25.537410 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:25.537432 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:25.537483 (+ 51us) server_negotiation.cc:407] Connection header received
0926 02:12:25.539233 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:25.539271 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:25.539884 (+ 613us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:25.540432 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:25.541725 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:25.543768 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:25.545301 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:25.546511 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:25.546723 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:25.551565 (+ 4842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:25.551584 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:25.551610 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:25.551674 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:25.555999 (+ 4325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:25.557155 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:25.557170 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:25.557177 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:25.557415 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:25.558013 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:25.558017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:25.558020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:25.558612 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:25.559042 (+ 430us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:25.559648 (+ 606us) server_negotiation.cc:299] Negotiation successful
0926 02:12:25.560006 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":347,"threads_started":1}
I20260926 02:12:25.566008 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:12:25.568140 7765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2218aa81bb41cbbb2579efad7c1a9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37499 } }
I20260926 02:12:25.568710 7765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:25.568850 7765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2218aa81bb41cbbb2579efad7c1a9b, State: Initialized, Role: FOLLOWER
I20260926 02:12:25.569523 7765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2218aa81bb41cbbb2579efad7c1a9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37499 } }
I20260926 02:12:25.569778 7765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:12:25.569949 7765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:12:25.570140 7765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:25.572844 7765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2218aa81bb41cbbb2579efad7c1a9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37499 } }
I20260926 02:12:25.573405 7765 leader_election.cc:302] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d2218aa81bb41cbbb2579efad7c1a9b; no voters:
I20260926 02:12:25.574039 7765 leader_election.cc:288] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:12:25.574261 7775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:25.576431 7765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:25.580147 7775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b [term 1 LEADER]: Becoming Leader. State: Replica: 9d2218aa81bb41cbbb2579efad7c1a9b, State: Running, Role: LEADER
I20260926 02:12:25.581029 7775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2218aa81bb41cbbb2579efad7c1a9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37499 } }
I20260926 02:12:25.584508 7776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d2218aa81bb41cbbb2579efad7c1a9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2218aa81bb41cbbb2579efad7c1a9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37499 } } }
I20260926 02:12:25.584996 7776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:25.585507 7777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2218aa81bb41cbbb2579efad7c1a9b. Latest consensus state: current_term: 1 leader_uuid: "9d2218aa81bb41cbbb2579efad7c1a9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2218aa81bb41cbbb2579efad7c1a9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37499 } } }
I20260926 02:12:25.585808 7777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:25.590253 7785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:25.596258 7785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:25.596627 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:25.571282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:39549 (local address 127.0.0.1:37499)
0926 02:12:25.571591 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:25.571606 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:25.572254 (+ 648us) server_negotiation.cc:407] Connection header received
0926 02:12:25.572524 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:25.572529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:25.572602 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:25.572729 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:25.575030 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:25.576928 (+ 1898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:25.579924 (+ 2996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:25.580772 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:25.580931 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:25.585039 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:25.585046 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:25.585051 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:25.585100 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:25.591446 (+ 6346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:25.592713 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:25.592725 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:25.592730 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:25.592932 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:25.593660 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:25.593666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:25.593670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:25.594149 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:25.594513 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:25.595890 (+ 1377us) server_negotiation.cc:299] Negotiation successful
0926 02:12:25.596112 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 02:12:25.609082 7785 catalog_manager.cc:1347] Generated new cluster ID: 53007529b3184f648bc961017ac1404d
I20260926 02:12:25.609234 7785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:25.634066 7785 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:25.635545 7785 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:25.650343 7785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d2218aa81bb41cbbb2579efad7c1a9b: Generated new TSK 0
I20260926 02:12:25.651564 7785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:12:25.658810 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 7698
I20260926 02:12:25.660280 7760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:25.761049 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 7698
W20260926 02:12:25.778481 7683 connection.cc:535] client connection to 127.0.0.1:37499 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37499): Connection reset by peer (error 104)
2026-09-26T02:12:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2062 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:12:25 dist-test-slave-03rn krb5kdc[7797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:12:25 dist-test-slave-03rn krb5kdc[7797](info): set up 2 sockets
Sep 26 02:12:25 dist-test-slave-03rn krb5kdc[7797](info): commencing operation
krb5kdc: starting...
W20260926 02:12:26.885058 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:12:27 dist-test-slave-03rn krb5kdc[7797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:12:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:12:27.155874 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39049
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:27.503189 7810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:27.503566 7810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:27.503746 7810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:27.511312 7810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:27.511477 7810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:27.511591 7810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:27.511698 7810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:27.511780 7810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:27.522100 7810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:27.523522 7810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:27.524917 7810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:27.537125 7816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:27.537077 7818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:27.537374 7815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:27.538065 7810 server_base.cc:1034] running on GCE node
I20260926 02:12:27.538983 7810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:27.540619 7810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:27.541903 7810 hybrid_clock.cc:648] HybridClock initialized: now 1790388747541861 us; error 58 us; skew 500 ppm
I20260926 02:12:27.542294 7810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:27 dist-test-slave-03rn krb5kdc[7797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:27.545670 7810 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:12:27.547261 7810 webserver.cc:460] Webserver started at http://127.0.0.1:40985/ using document root <none> and password file <none>
I20260926 02:12:27.547921 7810 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:27.548024 7810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:27.548389 7810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:27.551697 7810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "19cb50abb5f740d1948ed8e60fc37cdb"
format_stamp: "Formatted at 2026-09-26 02:12:27 on dist-test-slave-03rn"
server_key: "71ad233b1a4f4ecc8be5d221864f7b92"
server_key_iv: "17712feb285943fdc76ae4518a3c2153"
server_key_version: "encryptionkey@0"
I20260926 02:12:27.552564 7810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "19cb50abb5f740d1948ed8e60fc37cdb"
format_stamp: "Formatted at 2026-09-26 02:12:27 on dist-test-slave-03rn"
server_key: "71ad233b1a4f4ecc8be5d221864f7b92"
server_key_iv: "17712feb285943fdc76ae4518a3c2153"
server_key_version: "encryptionkey@0"
I20260926 02:12:27.558030 7810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:12:27.561880 7825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:27.563375 7810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:12:27.563541 7810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "19cb50abb5f740d1948ed8e60fc37cdb"
format_stamp: "Formatted at 2026-09-26 02:12:27 on dist-test-slave-03rn"
server_key: "71ad233b1a4f4ecc8be5d221864f7b92"
server_key_iv: "17712feb285943fdc76ae4518a3c2153"
server_key_version: "encryptionkey@0"
I20260926 02:12:27.563730 7810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:27.581632 7810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:27.586395 7810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:27.586640 7810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:27.606725 7810 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39049
I20260926 02:12:27.606793 7876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39049 every 8 connection(s)
I20260926 02:12:27.608799 7810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:27.612775 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 7810
I20260926 02:12:27.612980 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:27.612987 7877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:27.613533 19070 external_mini_cluster.cc:1428] Setting key 5b870911306564e6a1cff80bac6551b8
I20260926 02:12:27.626866 7877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb: Bootstrap starting.
I20260926 02:12:27.627297 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:27.615539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:51075 (local address 127.0.0.1:39049)
0926 02:12:27.616453 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:27.616465 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:27.616508 (+ 43us) server_negotiation.cc:407] Connection header received
0926 02:12:27.618221 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:27.618246 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:27.618694 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:27.619131 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:27.619782 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:27.621182 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:27.622566 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:27.623274 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:27.623507 (+ 233us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:27.624858 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:27.624866 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:27.624882 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:27.625491 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:27.625569 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:27.625594 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:27.625772 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:27.626052 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 02:12:27.626236 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":342,"threads_started":1}
I20260926 02:12:27.630909 7877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:27.631793 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:12:27.632238 7877 log.cc:824] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:27.635210 7877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb: No bootstrap required, opened a new log
I20260926 02:12:27.643601 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:27.636394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:50545 (local address 127.0.0.1:39049)
0926 02:12:27.636638 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:27.636646 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:27.637052 (+ 406us) server_negotiation.cc:407] Connection header received
0926 02:12:27.637270 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:27.637274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:27.637349 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:27.637437 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:27.638051 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:27.638846 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:27.640820 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:27.641304 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:27.641400 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:27.642702 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:27.642710 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:27.642713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:27.642736 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:27.642812 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:27.642832 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:27.642925 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:27.643137 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 02:12:27.643187 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:12:27.645541 7877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cb50abb5f740d1948ed8e60fc37cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39049 } }
I20260926 02:12:27.645989 7877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:27.646106 7877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19cb50abb5f740d1948ed8e60fc37cdb, State: Initialized, Role: FOLLOWER
I20260926 02:12:27.646824 7877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cb50abb5f740d1948ed8e60fc37cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39049 } }
I20260926 02:12:27.647131 7877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:12:27.647291 7877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:12:27.647482 7877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:27.650094 7877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cb50abb5f740d1948ed8e60fc37cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39049 } }
I20260926 02:12:27.650668 7877 leader_election.cc:302] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19cb50abb5f740d1948ed8e60fc37cdb; no voters:
I20260926 02:12:27.651387 7877 leader_election.cc:288] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:12:27.651551 7887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:27.652467 7887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb [term 1 LEADER]: Becoming Leader. State: Replica: 19cb50abb5f740d1948ed8e60fc37cdb, State: Running, Role: LEADER
I20260926 02:12:27.653023 7887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cb50abb5f740d1948ed8e60fc37cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39049 } }
I20260926 02:12:27.653833 7877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:27.656073 7889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19cb50abb5f740d1948ed8e60fc37cdb. Latest consensus state: current_term: 1 leader_uuid: "19cb50abb5f740d1948ed8e60fc37cdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cb50abb5f740d1948ed8e60fc37cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39049 } } }
I20260926 02:12:27.656662 7889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:27.656085 7888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19cb50abb5f740d1948ed8e60fc37cdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19cb50abb5f740d1948ed8e60fc37cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39049 } } }
I20260926 02:12:27.658349 7888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:27.659947 7894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:27.667577 7894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:27.677604 7894 catalog_manager.cc:1347] Generated new cluster ID: 79f37b18f3924c9ea234d75acffc05e7
I20260926 02:12:27.677771 7894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:27.705516 7894 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:27.707623 7894 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:27.718801 7894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19cb50abb5f740d1948ed8e60fc37cdb: Generated new TSK 0
I20260926 02:12:27.719907 7894 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:12:27.736012 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 7810
W20260926 02:12:27.737303 7872 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.229:51075
W20260926 02:12:27.738162 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 7810: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:12:27.738317 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 7810
W20260926 02:12:27.752640 7795 connection.cc:535] client connection to 127.0.0.1:39049 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39049): Connection reset by peer (error 104)
2026-09-26T02:12:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1973 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:12:27 dist-test-slave-03rn krb5kdc[7908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:12:27 dist-test-slave-03rn krb5kdc[7908](info): set up 2 sockets
Sep 26 02:12:27 dist-test-slave-03rn krb5kdc[7908](info): commencing operation
krb5kdc: starting...
W20260926 02:12:28.849123 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.007s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:12:29 dist-test-slave-03rn krb5kdc[7908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:12:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:12:29.123354 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39003
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:29.475376 7921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:29.475726 7921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:29.475924 7921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:29.484437 7921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:29.484565 7921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:29.484613 7921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:29.484679 7921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:29.484724 7921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:29.495266 7921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:29.496702 7921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:29.498008 7921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:29.510565 7927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:29.510653 7929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:29.510653 7926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:29.511639 7921 server_base.cc:1034] running on GCE node
I20260926 02:12:29.512602 7921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:29.514155 7921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:29.515385 7921 hybrid_clock.cc:648] HybridClock initialized: now 1790388749515338 us; error 40 us; skew 500 ppm
I20260926 02:12:29.515736 7921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:29 dist-test-slave-03rn krb5kdc[7908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:29.519230 7921 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:12:29.520994 7921 webserver.cc:460] Webserver started at http://127.0.0.1:38849/ using document root <none> and password file <none>
I20260926 02:12:29.521747 7921 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:29.521859 7921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:29.522159 7921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:29.525558 7921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "f862201ac97e472d85c17a98924ecaf0"
format_stamp: "Formatted at 2026-09-26 02:12:29 on dist-test-slave-03rn"
server_key: "55ac31b60f189dca0d9a11276d4df9b7"
server_key_iv: "4159417290ea8571c49288a0090dbdeb"
server_key_version: "encryptionkey@0"
I20260926 02:12:29.526396 7921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "f862201ac97e472d85c17a98924ecaf0"
format_stamp: "Formatted at 2026-09-26 02:12:29 on dist-test-slave-03rn"
server_key: "55ac31b60f189dca0d9a11276d4df9b7"
server_key_iv: "4159417290ea8571c49288a0090dbdeb"
server_key_version: "encryptionkey@0"
I20260926 02:12:29.532377 7921 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 02:12:29.536574 7936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:29.538225 7921 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:12:29.538453 7921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "f862201ac97e472d85c17a98924ecaf0"
format_stamp: "Formatted at 2026-09-26 02:12:29 on dist-test-slave-03rn"
server_key: "55ac31b60f189dca0d9a11276d4df9b7"
server_key_iv: "4159417290ea8571c49288a0090dbdeb"
server_key_version: "encryptionkey@0"
I20260926 02:12:29.538699 7921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:29.544955 7921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:29.549738 7921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:29.550078 7921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:29.570305 7921 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39003
I20260926 02:12:29.570416 7987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39003 every 8 connection(s)
I20260926 02:12:29.571849 7921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:29.575950 7988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:29.581131 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 7921
I20260926 02:12:29.581283 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:29.581713 19070 external_mini_cluster.cc:1428] Setting key 7f861b9c2532b7e027b03b0d4767d39d
I20260926 02:12:29.589119 7988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0: Bootstrap starting.
I20260926 02:12:29.591559 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:29.583768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:38013 (local address 127.0.0.1:39003)
0926 02:12:29.584637 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:29.584655 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:29.584713 (+ 58us) server_negotiation.cc:407] Connection header received
0926 02:12:29.587293 (+ 2580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:29.587328 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:29.587844 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:29.588372 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:29.589435 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:29.589445 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:29.589467 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:29.589486 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:29.589572 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:29.589602 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:29.589846 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:29.589977 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 02:12:29.590186 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":363,"threads_started":1}
I20260926 02:12:29.593379 7988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:29.594664 7988 log.cc:824] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:29.595963 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:12:29.598762 7988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0: No bootstrap required, opened a new log
I20260926 02:12:29.604385 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:29.601726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:55619 (local address 127.0.0.1:39003)
0926 02:12:29.601956 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:29.601963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:29.602317 (+ 354us) server_negotiation.cc:407] Connection header received
0926 02:12:29.602469 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:29.602475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:29.602546 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:29.602664 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:29.603613 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:29.603619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:29.603624 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:29.603641 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:29.603691 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:29.603710 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:29.603836 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:29.603949 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 02:12:29.604011 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260926 02:12:29.608654 7988 raft_consensus.cc:348] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862201ac97e472d85c17a98924ecaf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39003 } }
I20260926 02:12:29.609067 7988 raft_consensus.cc:374] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:29.609174 7988 raft_consensus.cc:729] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f862201ac97e472d85c17a98924ecaf0, State: Initialized, Role: FOLLOWER
I20260926 02:12:29.609848 7988 consensus_queue.cc:260] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862201ac97e472d85c17a98924ecaf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39003 } }
I20260926 02:12:29.610100 7988 raft_consensus.cc:388] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:12:29.610275 7988 raft_consensus.cc:482] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:12:29.610466 7988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:29.613380 7988 raft_consensus.cc:504] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862201ac97e472d85c17a98924ecaf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39003 } }
I20260926 02:12:29.614002 7988 leader_election.cc:302] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f862201ac97e472d85c17a98924ecaf0; no voters:
I20260926 02:12:29.614778 7988 leader_election.cc:288] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:12:29.614987 7998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:29.615746 7998 raft_consensus.cc:686] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0 [term 1 LEADER]: Becoming Leader. State: Replica: f862201ac97e472d85c17a98924ecaf0, State: Running, Role: LEADER
I20260926 02:12:29.616508 7998 consensus_queue.cc:237] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862201ac97e472d85c17a98924ecaf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39003 } }
I20260926 02:12:29.617933 7988 sys_catalog.cc:564] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:29.619894 7999 sys_catalog.cc:455] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f862201ac97e472d85c17a98924ecaf0. Latest consensus state: current_term: 1 leader_uuid: "f862201ac97e472d85c17a98924ecaf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862201ac97e472d85c17a98924ecaf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39003 } } }
I20260926 02:12:29.619962 8000 sys_catalog.cc:455] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f862201ac97e472d85c17a98924ecaf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862201ac97e472d85c17a98924ecaf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39003 } } }
I20260926 02:12:29.620427 7999 sys_catalog.cc:458] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:29.620594 8000 sys_catalog.cc:458] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:29.622258 8003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:29.628350 8003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:29.638275 8003 catalog_manager.cc:1347] Generated new cluster ID: 6bc22ba76f0f4933a47872d1e692e618
I20260926 02:12:29.638582 8003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:29.653432 8003 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:29.653992 8003 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:29.665058 8003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f862201ac97e472d85c17a98924ecaf0: Generated new TSK 0
I20260926 02:12:29.666191 8003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:12:29.675344 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 7921
W20260926 02:12:29.676693 7983 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.229:38013
W20260926 02:12:29.677541 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 7921: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:12:29.677712 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 7921
2026-09-26T02:12:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1939 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:12:29 dist-test-slave-03rn krb5kdc[8019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:12:29 dist-test-slave-03rn krb5kdc[8019](info): set up 2 sockets
Sep 26 02:12:29 dist-test-slave-03rn krb5kdc[8019](info): commencing operation
krb5kdc: starting...
W20260926 02:12:30.797104 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:12:30 dist-test-slave-03rn krb5kdc[8019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:12:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:12:31.079609 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:31.445655 8032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:31.445962 8032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:31.446056 8032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:31.453733 8032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:31.453874 8032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:31.453923 8032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:31.453979 8032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:31.454025 8032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:31.464663 8032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44351
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:31.465987 8032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:31.467447 8032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:31.479852 8038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:31.479887 8040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:31.479770 8037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:31.481134 8032 server_base.cc:1034] running on GCE node
I20260926 02:12:31.481948 8032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:31.483492 8032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:31.484740 8032 hybrid_clock.cc:648] HybridClock initialized: now 1790388751484648 us; error 65 us; skew 500 ppm
I20260926 02:12:31.485090 8032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:31 dist-test-slave-03rn krb5kdc[8019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:31.488571 8032 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:12:31.490180 8032 webserver.cc:460] Webserver started at http://127.0.0.1:45581/ using document root <none> and password file <none>
I20260926 02:12:31.490926 8032 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:31.491079 8032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:31.491390 8032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:31.494599 8032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "db2cd368a9f9415fb2b32b277ce8ee8a"
format_stamp: "Formatted at 2026-09-26 02:12:31 on dist-test-slave-03rn"
server_key: "444acc4bc49c9863cb9f0569592f9f99"
server_key_iv: "eed33677346e1a08b5c481189532612c"
server_key_version: "encryptionkey@0"
I20260926 02:12:31.495471 8032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "db2cd368a9f9415fb2b32b277ce8ee8a"
format_stamp: "Formatted at 2026-09-26 02:12:31 on dist-test-slave-03rn"
server_key: "444acc4bc49c9863cb9f0569592f9f99"
server_key_iv: "eed33677346e1a08b5c481189532612c"
server_key_version: "encryptionkey@0"
I20260926 02:12:31.501125 8032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:12:31.504916 8047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:31.506376 8032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:12:31.506590 8032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "db2cd368a9f9415fb2b32b277ce8ee8a"
format_stamp: "Formatted at 2026-09-26 02:12:31 on dist-test-slave-03rn"
server_key: "444acc4bc49c9863cb9f0569592f9f99"
server_key_iv: "eed33677346e1a08b5c481189532612c"
server_key_version: "encryptionkey@0"
I20260926 02:12:31.506816 8032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:31.528357 8032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:31.533192 8032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:31.533525 8032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:31.553269 8032 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44351
I20260926 02:12:31.553315 8098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44351 every 8 connection(s)
I20260926 02:12:31.554824 8032 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:31.558151 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 8032
I20260926 02:12:31.558372 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:31.558907 19070 external_mini_cluster.cc:1428] Setting key 6e60e661eeb6b249e1b52f437305b5b3
I20260926 02:12:31.559274 8099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:31.575387 8099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a: Bootstrap starting.
I20260926 02:12:31.579698 8099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:31.581159 8099 log.cc:824] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:12:31 dist-test-slave-03rn krb5kdc[8019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 02:12:31.584417 8099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a: No bootstrap required, opened a new log
I20260926 02:12:31.591035 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:31.564260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:56137 (local address 127.0.0.1:44351)
0926 02:12:31.566784 (+ 2524us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:31.566805 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:31.566871 (+ 66us) server_negotiation.cc:407] Connection header received
0926 02:12:31.569361 (+ 2490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:31.569396 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:31.570006 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:31.570512 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:31.573067 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:31.574635 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:31.576523 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:31.577273 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:31.577435 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:31.582724 (+ 5289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:31.582734 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:31.582761 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:31.582821 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:31.586169 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:31.587189 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:31.587204 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:31.587215 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:31.587418 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:31.587942 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:31.587946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:31.587948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:31.588636 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:31.589034 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:31.589545 (+ 511us) server_negotiation.cc:299] Negotiation successful
0926 02:12:31.589906 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2226,"thread_start_us":1933,"threads_started":1}
I20260926 02:12:31.592813 8099 raft_consensus.cc:348] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db2cd368a9f9415fb2b32b277ce8ee8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44351 } }
I20260926 02:12:31.593231 8099 raft_consensus.cc:374] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:31.593357 8099 raft_consensus.cc:729] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db2cd368a9f9415fb2b32b277ce8ee8a, State: Initialized, Role: FOLLOWER
I20260926 02:12:31.593999 8099 consensus_queue.cc:260] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db2cd368a9f9415fb2b32b277ce8ee8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44351 } }
I20260926 02:12:31.594259 8099 raft_consensus.cc:388] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:12:31.594413 8099 raft_consensus.cc:482] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:12:31.594625 8099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:31.595415 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:12:31.597189 8099 raft_consensus.cc:504] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db2cd368a9f9415fb2b32b277ce8ee8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44351 } }
I20260926 02:12:31.597746 8099 leader_election.cc:302] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db2cd368a9f9415fb2b32b277ce8ee8a; no voters:
I20260926 02:12:31.598501 8099 leader_election.cc:288] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:12:31.598673 8106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:31.600829 8099 sys_catalog.cc:564] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:31.602835 8106 raft_consensus.cc:686] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a [term 1 LEADER]: Becoming Leader. State: Replica: db2cd368a9f9415fb2b32b277ce8ee8a, State: Running, Role: LEADER
I20260926 02:12:31.603626 8106 consensus_queue.cc:237] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db2cd368a9f9415fb2b32b277ce8ee8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44351 } }
I20260926 02:12:31.607301 8109 sys_catalog.cc:455] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db2cd368a9f9415fb2b32b277ce8ee8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db2cd368a9f9415fb2b32b277ce8ee8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44351 } } }
I20260926 02:12:31.607723 8109 sys_catalog.cc:458] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:31.612808 8110 sys_catalog.cc:455] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader db2cd368a9f9415fb2b32b277ce8ee8a. Latest consensus state: current_term: 1 leader_uuid: "db2cd368a9f9415fb2b32b277ce8ee8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db2cd368a9f9415fb2b32b277ce8ee8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44351 } } }
I20260926 02:12:31.613322 8110 sys_catalog.cc:458] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:31.618152 8116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:31.624543 8116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:31.638056 8116 catalog_manager.cc:1347] Generated new cluster ID: 08f571d657d94276b176fa72323f780d
I20260926 02:12:31.638178 8116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:31.641083 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:31.607345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:44279 (local address 127.0.0.1:44351)
0926 02:12:31.607586 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:31.607593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:31.618157 (+ 10564us) server_negotiation.cc:407] Connection header received
0926 02:12:31.618426 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:31.618432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:31.618523 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:31.618650 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:31.621099 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:31.622005 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:31.627343 (+ 5338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:31.628442 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:31.628595 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:31.633230 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:31.633242 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:31.633250 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:31.633324 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:31.637636 (+ 4312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:31.638739 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:31.638744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:31.638746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:31.638878 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:31.639461 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:31.639464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:31.639466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:31.639788 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:31.639987 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:31.640485 (+ 498us) server_negotiation.cc:299] Negotiation successful
0926 02:12:31.640670 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":75}
I20260926 02:12:31.667907 8116 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:31.669639 8116 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:31.677640 8116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db2cd368a9f9415fb2b32b277ce8ee8a: Generated new TSK 0
I20260926 02:12:31.678579 8116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:12:31.692308 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 8032
I20260926 02:12:31.693632 8094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:31.789901 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 8032
W20260926 02:12:31.807296 8017 connection.cc:535] client connection to 127.0.0.1:44351 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44351): Connection reset by peer (error 104)
2026-09-26T02:12:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2119 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:12:31 dist-test-slave-03rn krb5kdc[8131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:12:31 dist-test-slave-03rn krb5kdc[8131](info): set up 2 sockets
Sep 26 02:12:31 dist-test-slave-03rn krb5kdc[8131](info): commencing operation
krb5kdc: starting...
W20260926 02:12:32.925238 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:12:33 dist-test-slave-03rn krb5kdc[8131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:12:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:12:33.196262 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39913
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:33.546605 8144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:33.546922 8144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:33.547017 8144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:33.554534 8144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:33.554667 8144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:33.554716 8144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:33.554770 8144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:33.554816 8144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:33.565189 8144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39913
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:33.566478 8144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:33.567725 8144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:33.579854 8149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:33.579852 8150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:33.579872 8152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:33.580690 8144 server_base.cc:1034] running on GCE node
I20260926 02:12:33.581522 8144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:33.583083 8144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:33.584306 8144 hybrid_clock.cc:648] HybridClock initialized: now 1790388753584261 us; error 45 us; skew 500 ppm
I20260926 02:12:33.584681 8144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:33 dist-test-slave-03rn krb5kdc[8131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:33.587901 8144 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:12:33.589478 8144 webserver.cc:460] Webserver started at http://127.0.0.1:41009/ using document root <none> and password file <none>
I20260926 02:12:33.590152 8144 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:33.590256 8144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:33.590533 8144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:33.593730 8144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "0376f666ef02494b88a4ffe0126e97ab"
format_stamp: "Formatted at 2026-09-26 02:12:33 on dist-test-slave-03rn"
server_key: "4fbb0ab0492bea858c86b5f3666aba0d"
server_key_iv: "a82d9bd6b8a08fa912423b9ff873803f"
server_key_version: "encryptionkey@0"
I20260926 02:12:33.594476 8144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "0376f666ef02494b88a4ffe0126e97ab"
format_stamp: "Formatted at 2026-09-26 02:12:33 on dist-test-slave-03rn"
server_key: "4fbb0ab0492bea858c86b5f3666aba0d"
server_key_iv: "a82d9bd6b8a08fa912423b9ff873803f"
server_key_version: "encryptionkey@0"
I20260926 02:12:33.600054 8144 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:12:33.603986 8159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:33.605574 8144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:12:33.605731 8144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "0376f666ef02494b88a4ffe0126e97ab"
format_stamp: "Formatted at 2026-09-26 02:12:33 on dist-test-slave-03rn"
server_key: "4fbb0ab0492bea858c86b5f3666aba0d"
server_key_iv: "a82d9bd6b8a08fa912423b9ff873803f"
server_key_version: "encryptionkey@0"
I20260926 02:12:33.605912 8144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:33.618315 8144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:33.622552 8144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:33.622795 8144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:33.642300 8210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39913 every 8 connection(s)
I20260926 02:12:33.642330 8144 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39913
I20260926 02:12:33.644207 8144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:33.648312 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:33.653257 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 8144
I20260926 02:12:33.653474 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:33.653892 19070 external_mini_cluster.cc:1428] Setting key 6591209a6301c0afa6ac9fd94c409027
I20260926 02:12:33.661681 8211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab: Bootstrap starting.
I20260926 02:12:33.665825 8211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:33.667333 8211 log.cc:824] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:33.671182 8211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab: No bootstrap required, opened a new log
I20260926 02:12:33.671270 8214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:33.655893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:37935 (local address 127.0.0.1:39913)
0926 02:12:33.656810 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:33.656833 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:33.656888 (+ 55us) server_negotiation.cc:407] Connection header received
0926 02:12:33.659619 (+ 2731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:33.659657 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:33.660297 (+ 640us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:33.660786 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:33.661386 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:33.663387 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:33.665445 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:33.666647 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:33.666864 (+ 217us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:33.668314 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:33.668324 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:33.668340 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:33.669057 (+ 717us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:33.669165 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:33.669204 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:33.669383 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:33.669702 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 02:12:33.669898 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":403,"threads_started":1}
I20260926 02:12:33.676070 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:12:33.680526 8211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0376f666ef02494b88a4ffe0126e97ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39913 } }
I20260926 02:12:33.681155 8211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:33.681324 8211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0376f666ef02494b88a4ffe0126e97ab, State: Initialized, Role: FOLLOWER
I20260926 02:12:33.682128 8211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0376f666ef02494b88a4ffe0126e97ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39913 } }
I20260926 02:12:33.682502 8211 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:12:33.682668 8211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:12:33.682870 8211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:33.686009 8211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0376f666ef02494b88a4ffe0126e97ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39913 } }
I20260926 02:12:33.686753 8211 leader_election.cc:302] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0376f666ef02494b88a4ffe0126e97ab; no voters:
I20260926 02:12:33.687500 8211 leader_election.cc:288] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:12:33.687656 8221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:33.689208 8214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:33.680843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:42949 (local address 127.0.0.1:39913)
0926 02:12:33.681054 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:33.681062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:33.681755 (+ 693us) server_negotiation.cc:407] Connection header received
0926 02:12:33.681990 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:33.681995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:33.682078 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:33.682199 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:33.682913 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:33.684097 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:33.685579 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:33.686308 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:33.686446 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:33.687944 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:33.687952 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:33.687961 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:33.688007 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:33.688110 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:33.688132 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:33.688725 (+ 593us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:33.688800 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 02:12:33.688855 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 02:12:33.690058 8211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:33.692518 8221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab [term 1 LEADER]: Becoming Leader. State: Replica: 0376f666ef02494b88a4ffe0126e97ab, State: Running, Role: LEADER
I20260926 02:12:33.693141 8221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0376f666ef02494b88a4ffe0126e97ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39913 } }
I20260926 02:12:33.696779 8223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0376f666ef02494b88a4ffe0126e97ab. Latest consensus state: current_term: 1 leader_uuid: "0376f666ef02494b88a4ffe0126e97ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0376f666ef02494b88a4ffe0126e97ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39913 } } }
I20260926 02:12:33.697647 8223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:33.697232 8222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0376f666ef02494b88a4ffe0126e97ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0376f666ef02494b88a4ffe0126e97ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39913 } } }
I20260926 02:12:33.698247 8222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:33.700726 8232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:33.706324 8232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:33.717219 8232 catalog_manager.cc:1347] Generated new cluster ID: 771f0e5692d9484fbd626c90a8af54d3
I20260926 02:12:33.717391 8232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:33.736238 8232 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:33.738312 8232 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:33.757962 8232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0376f666ef02494b88a4ffe0126e97ab: Generated new TSK 0
I20260926 02:12:33.759184 8232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:12:33.776364 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 8144
W20260926 02:12:33.777808 8206 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.229:37935
W20260926 02:12:33.778687 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 8144: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:12:33.778877 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 8144
W20260926 02:12:33.796223 8129 connection.cc:535] client connection to 127.0.0.1:39913 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39913): Connection reset by peer (error 104)
2026-09-26T02:12:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1984 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:12:33 dist-test-slave-03rn krb5kdc[8242](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:12:33 dist-test-slave-03rn krb5kdc[8242](info): set up 2 sockets
Sep 26 02:12:33 dist-test-slave-03rn krb5kdc[8242](info): commencing operation
krb5kdc: starting...
W20260926 02:12:34.900985 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:12:35 dist-test-slave-03rn krb5kdc[8242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:12:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:12:35.169865 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43333
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:35.521798 8255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:35.522185 8255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:35.522351 8255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:35.531252 8255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:35.531486 8255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:35.531610 8255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:35.531724 8255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:35.531804 8255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:35.542629 8255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43333
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:35.544080 8255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:35.545552 8255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:35.558529 8263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:35.558691 8260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:35.558446 8261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:35.559806 8255 server_base.cc:1034] running on GCE node
I20260926 02:12:35.560804 8255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:35.562493 8255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:35.563791 8255 hybrid_clock.cc:648] HybridClock initialized: now 1790388755563756 us; error 61 us; skew 500 ppm
I20260926 02:12:35.564306 8255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:35 dist-test-slave-03rn krb5kdc[8242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:35.568017 8255 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:12:35.569905 8255 webserver.cc:460] Webserver started at http://127.0.0.1:46647/ using document root <none> and password file <none>
I20260926 02:12:35.570729 8255 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:35.570905 8255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:35.571269 8255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:35.574872 8255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "79c9c1ac72714fb3af6fb63573b4e87b"
format_stamp: "Formatted at 2026-09-26 02:12:35 on dist-test-slave-03rn"
server_key: "e9e1328d8159035e5830c02a53e7353c"
server_key_iv: "736152f32d24725c5893e316abd06032"
server_key_version: "encryptionkey@0"
I20260926 02:12:35.575824 8255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "79c9c1ac72714fb3af6fb63573b4e87b"
format_stamp: "Formatted at 2026-09-26 02:12:35 on dist-test-slave-03rn"
server_key: "e9e1328d8159035e5830c02a53e7353c"
server_key_iv: "736152f32d24725c5893e316abd06032"
server_key_version: "encryptionkey@0"
I20260926 02:12:35.582181 8255 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 02:12:35.586387 8270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:35.587950 8255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:12:35.588232 8255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "79c9c1ac72714fb3af6fb63573b4e87b"
format_stamp: "Formatted at 2026-09-26 02:12:35 on dist-test-slave-03rn"
server_key: "e9e1328d8159035e5830c02a53e7353c"
server_key_iv: "736152f32d24725c5893e316abd06032"
server_key_version: "encryptionkey@0"
I20260926 02:12:35.588480 8255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:35.595072 8255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:35.600034 8255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:35.600416 8255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:35.621240 8255 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43333
I20260926 02:12:35.621279 8321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43333 every 8 connection(s)
I20260926 02:12:35.622843 8255 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:35.627334 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:35.627353 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 8255
I20260926 02:12:35.627516 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:35.628054 19070 external_mini_cluster.cc:1428] Setting key c3cb18a7ab732974721aea0079cd1f16
I20260926 02:12:35.636619 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:35.630092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:44039 (local address 127.0.0.1:43333)
0926 02:12:35.630986 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:35.630999 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:35.631042 (+ 43us) server_negotiation.cc:407] Connection header received
0926 02:12:35.632819 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:35.632845 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:35.633240 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:35.633704 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:35.634835 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:35.634841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:35.634856 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:35.634870 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:35.634929 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:35.634951 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:35.635208 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:35.635368 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 02:12:35.635569 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":373,"threads_started":1}
I20260926 02:12:35.641112 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:12:35.642765 8322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b: Bootstrap starting.
I20260926 02:12:35.646642 8322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:35.648262 8322 log.cc:824] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:35.648567 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:35.645784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:40113 (local address 127.0.0.1:43333)
0926 02:12:35.645982 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:35.645991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:35.646354 (+ 363us) server_negotiation.cc:407] Connection header received
0926 02:12:35.646578 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:35.646583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:35.646638 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:35.646750 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:35.647784 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:35.647789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:35.647793 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:35.647810 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:35.647859 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:35.647874 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:35.648021 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:35.648186 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 02:12:35.648264 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:12:35.652626 8322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b: No bootstrap required, opened a new log
I20260926 02:12:35.661495 8322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c9c1ac72714fb3af6fb63573b4e87b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43333 } }
I20260926 02:12:35.662002 8322 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:35.662128 8322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79c9c1ac72714fb3af6fb63573b4e87b, State: Initialized, Role: FOLLOWER
I20260926 02:12:35.662817 8322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c9c1ac72714fb3af6fb63573b4e87b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43333 } }
I20260926 02:12:35.663077 8322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:12:35.663254 8322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:12:35.663448 8322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:35.665987 8322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c9c1ac72714fb3af6fb63573b4e87b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43333 } }
I20260926 02:12:35.666621 8322 leader_election.cc:302] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79c9c1ac72714fb3af6fb63573b4e87b; no voters:
I20260926 02:12:35.667389 8322 leader_election.cc:288] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:12:35.667587 8332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:35.668363 8332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b [term 1 LEADER]: Becoming Leader. State: Replica: 79c9c1ac72714fb3af6fb63573b4e87b, State: Running, Role: LEADER
I20260926 02:12:35.668907 8332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c9c1ac72714fb3af6fb63573b4e87b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43333 } }
I20260926 02:12:35.669878 8322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:35.672056 8334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79c9c1ac72714fb3af6fb63573b4e87b. Latest consensus state: current_term: 1 leader_uuid: "79c9c1ac72714fb3af6fb63573b4e87b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c9c1ac72714fb3af6fb63573b4e87b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43333 } } }
I20260926 02:12:35.673012 8334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:35.672436 8333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79c9c1ac72714fb3af6fb63573b4e87b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c9c1ac72714fb3af6fb63573b4e87b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43333 } } }
I20260926 02:12:35.673975 8333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:35.674204 8340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:35.680658 8340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:35.691268 8340 catalog_manager.cc:1347] Generated new cluster ID: a234a32e01f14a0b99b49669c193f53e
I20260926 02:12:35.691431 8340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:35.716213 8340 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:35.716790 8340 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:35.734853 8340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79c9c1ac72714fb3af6fb63573b4e87b: Generated new TSK 0
I20260926 02:12:35.736032 8340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:12:35.754110 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 8255
W20260926 02:12:35.755394 8317 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.229:44039
W20260926 02:12:35.756505 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 8255: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:12:35.756688 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 8255
2026-09-26T02:12:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1977 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:12:35 dist-test-slave-03rn krb5kdc[8353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:12:35 dist-test-slave-03rn krb5kdc[8353](info): set up 2 sockets
Sep 26 02:12:35 dist-test-slave-03rn krb5kdc[8353](info): commencing operation
krb5kdc: starting...
W20260926 02:12:36.888959 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:12:37 dist-test-slave-03rn krb5kdc[8353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:12:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:12:37.155903 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:38507
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:38507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:37.498970 8366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:37.499307 8366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:37.499402 8366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:37.506956 8366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:37.507091 8366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:37.507145 8366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:37.507208 8366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:37.507253 8366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:37.517625 8366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:38507
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:38507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:37.519134 8366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:37.520787 8366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:37.533023 8372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:37.532979 8371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:37.533062 8374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:37.534041 8366 server_base.cc:1034] running on GCE node
I20260926 02:12:37.534883 8366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:37.536418 8366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:37.537652 8366 hybrid_clock.cc:648] HybridClock initialized: now 1790388757537600 us; error 45 us; skew 500 ppm
I20260926 02:12:37.538050 8366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:37 dist-test-slave-03rn krb5kdc[8353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388757, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:37.541253 8366 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:12:37.542845 8366 webserver.cc:460] Webserver started at http://127.18.159.190:34267/ using document root <none> and password file <none>
I20260926 02:12:37.543576 8366 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:37.543735 8366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:37.544055 8366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:37.547236 8366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "b152ee70b89548c095ddb228e81b115c"
format_stamp: "Formatted at 2026-09-26 02:12:37 on dist-test-slave-03rn"
server_key: "7cd334e8ac5769d0988a14064249b4ce"
server_key_iv: "58d77575ddc7f2536dd8d6eae9bfd51e"
server_key_version: "encryptionkey@0"
I20260926 02:12:37.548069 8366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "b152ee70b89548c095ddb228e81b115c"
format_stamp: "Formatted at 2026-09-26 02:12:37 on dist-test-slave-03rn"
server_key: "7cd334e8ac5769d0988a14064249b4ce"
server_key_iv: "58d77575ddc7f2536dd8d6eae9bfd51e"
server_key_version: "encryptionkey@0"
I20260926 02:12:37.553634 8366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:12:37.557404 8381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:37.558856 8366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:12:37.559067 8366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "b152ee70b89548c095ddb228e81b115c"
format_stamp: "Formatted at 2026-09-26 02:12:37 on dist-test-slave-03rn"
server_key: "7cd334e8ac5769d0988a14064249b4ce"
server_key_iv: "58d77575ddc7f2536dd8d6eae9bfd51e"
server_key_version: "encryptionkey@0"
I20260926 02:12:37.559303 8366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:37.586095 8366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:37.590337 8366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:37.590646 8366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:37.610220 8366 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:38507
I20260926 02:12:37.610258 8432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:38507 every 8 connection(s)
I20260926 02:12:37.611734 8366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:37.612629 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 8366
I20260926 02:12:37.612847 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:37.613255 19070 external_mini_cluster.cc:1428] Setting key 56f91ec2867d43fab2a03e2c68639ee4
I20260926 02:12:37.616559 8433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:12:37 dist-test-slave-03rn krb5kdc[8353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:12:37.630467 8433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c: Bootstrap starting.
I20260926 02:12:37.634842 8433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:37.636264 8433 log.cc:824] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:37.638903 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:37.615185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34777 (local address 127.18.159.190:38507)
0926 02:12:37.615945 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:37.615957 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:37.615999 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:12:37.617824 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:37.617846 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:37.618239 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:37.618558 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:37.620415 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:37.621785 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:37.623590 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:37.624379 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:37.624511 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:37.629584 (+ 5073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:37.629595 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:37.629617 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:37.629676 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:37.633801 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:37.634768 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:37.634778 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:37.634787 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:37.635042 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:37.635532 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:37.635537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:37.635540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:37.636354 (+ 814us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:37.636649 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:37.637161 (+ 512us) server_negotiation.cc:299] Negotiation successful
0926 02:12:37.637571 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":321,"threads_started":1}
I20260926 02:12:37.639932 8433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c: No bootstrap required, opened a new log
I20260926 02:12:37.643029 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:12:37.648783 8433 raft_consensus.cc:348] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b152ee70b89548c095ddb228e81b115c" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38507 } }
I20260926 02:12:37.649344 8433 raft_consensus.cc:374] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:37.649456 8433 raft_consensus.cc:729] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b152ee70b89548c095ddb228e81b115c, State: Initialized, Role: FOLLOWER
I20260926 02:12:37.650205 8433 consensus_queue.cc:260] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b152ee70b89548c095ddb228e81b115c" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38507 } }
I20260926 02:12:37.650519 8433 raft_consensus.cc:388] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:12:37.650698 8433 raft_consensus.cc:482] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:12:37.650918 8433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:37.654161 8433 raft_consensus.cc:504] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b152ee70b89548c095ddb228e81b115c" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38507 } }
I20260926 02:12:37.654870 8433 leader_election.cc:302] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b152ee70b89548c095ddb228e81b115c; no voters:
I20260926 02:12:37.655656 8433 leader_election.cc:288] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:12:37.655857 8443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:37.659375 8443 raft_consensus.cc:686] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c [term 1 LEADER]: Becoming Leader. State: Replica: b152ee70b89548c095ddb228e81b115c, State: Running, Role: LEADER
I20260926 02:12:37.660235 8443 consensus_queue.cc:237] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b152ee70b89548c095ddb228e81b115c" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38507 } }
I20260926 02:12:37.663136 8433 sys_catalog.cc:564] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:37.663875 8444 sys_catalog.cc:455] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b152ee70b89548c095ddb228e81b115c. Latest consensus state: current_term: 1 leader_uuid: "b152ee70b89548c095ddb228e81b115c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b152ee70b89548c095ddb228e81b115c" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38507 } } }
I20260926 02:12:37.664372 8444 sys_catalog.cc:458] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:37.666972 8445 sys_catalog.cc:455] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b152ee70b89548c095ddb228e81b115c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b152ee70b89548c095ddb228e81b115c" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 38507 } } }
I20260926 02:12:37.667474 8448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:37.667476 8445 sys_catalog.cc:458] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:37.669010 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:37.647469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42467 (local address 127.18.159.190:38507)
0926 02:12:37.647703 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:37.647712 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:37.648250 (+ 538us) server_negotiation.cc:407] Connection header received
0926 02:12:37.648455 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:37.648459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:37.648524 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:37.648606 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:37.650584 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:37.651580 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:37.653535 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:37.653959 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:37.654054 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:37.657966 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:37.657977 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:37.657986 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:37.658052 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:37.663344 (+ 5292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:37.664779 (+ 1435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:37.664789 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:37.664795 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:37.665056 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:37.665845 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:37.665853 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:37.665860 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:37.666363 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:37.666799 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:37.668234 (+ 1435us) server_negotiation.cc:299] Negotiation successful
0926 02:12:37.668481 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:12:37.674048 8448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:37.685218 8448 catalog_manager.cc:1347] Generated new cluster ID: 96c8f6a584e34a1598497b40336ea305
I20260926 02:12:37.685366 8448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:37.700110 8448 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:37.701630 8448 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:37.708877 8448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b152ee70b89548c095ddb228e81b115c: Generated new TSK 0
I20260926 02:12:37.709817 8448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:12:37.719797 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 8366
I20260926 02:12:37.721192 8428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:37.813594 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 8366
W20260926 02:12:37.830632 8351 connection.cc:535] client connection to 127.18.159.190:38507 recv error: Network error: failed to read from TLS socket (remote: 127.18.159.190:38507): Connection reset by peer (error 104)
2026-09-26T02:12:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2056 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:12:37 dist-test-slave-03rn krb5kdc[8465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:12:37 dist-test-slave-03rn krb5kdc[8465](info): set up 2 sockets
Sep 26 02:12:37 dist-test-slave-03rn krb5kdc[8465](info): commencing operation
krb5kdc: starting...
W20260926 02:12:38.924984 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:12:39 dist-test-slave-03rn krb5kdc[8465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:12:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:12:39.197141 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:39859
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:39859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:39.546622 8478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:39.547009 8478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:39.547202 8478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:39.554870 8478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:39.555053 8478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:39.555186 8478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:39.555297 8478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:39.555377 8478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:39.565814 8478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:39859
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:39859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:39.567239 8478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:39.568665 8478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:39.581414 8484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:39.581362 8486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:39.581429 8483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:39.582208 8478 server_base.cc:1034] running on GCE node
I20260926 02:12:39.583299 8478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:39.584952 8478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:39.586247 8478 hybrid_clock.cc:648] HybridClock initialized: now 1790388759586145 us; error 53 us; skew 500 ppm
I20260926 02:12:39.586695 8478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:39 dist-test-slave-03rn krb5kdc[8465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388759, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:39.590224 8478 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:12:39.591890 8478 webserver.cc:460] Webserver started at http://127.18.159.190:35197/ using document root <none> and password file <none>
I20260926 02:12:39.592707 8478 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:39.592859 8478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:39.593184 8478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:39.596498 8478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "a8c1f95d4a6b426caddc310647fdc64d"
format_stamp: "Formatted at 2026-09-26 02:12:39 on dist-test-slave-03rn"
server_key: "4011d50867b97b97dbddb8a5b3ba6dd9"
server_key_iv: "0e0f0bef2cbe3d6a6405e54cc04a6a45"
server_key_version: "encryptionkey@0"
I20260926 02:12:39.597342 8478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "a8c1f95d4a6b426caddc310647fdc64d"
format_stamp: "Formatted at 2026-09-26 02:12:39 on dist-test-slave-03rn"
server_key: "4011d50867b97b97dbddb8a5b3ba6dd9"
server_key_iv: "0e0f0bef2cbe3d6a6405e54cc04a6a45"
server_key_version: "encryptionkey@0"
I20260926 02:12:39.602993 8478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:12:39.606884 8493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:39.608559 8478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:12:39.608767 8478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "a8c1f95d4a6b426caddc310647fdc64d"
format_stamp: "Formatted at 2026-09-26 02:12:39 on dist-test-slave-03rn"
server_key: "4011d50867b97b97dbddb8a5b3ba6dd9"
server_key_iv: "0e0f0bef2cbe3d6a6405e54cc04a6a45"
server_key_version: "encryptionkey@0"
I20260926 02:12:39.608978 8478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:39.636044 8478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:39.640496 8478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:39.640825 8478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:39.660312 8478 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:39859
I20260926 02:12:39.660346 8544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:39859 every 8 connection(s)
I20260926 02:12:39.661782 8478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:39.663905 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 8478
I20260926 02:12:39.664090 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:39.664673 19070 external_mini_cluster.cc:1428] Setting key 6a3bff224d9351bdf1f7928f999047f3
I20260926 02:12:39.666584 8545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:39.681010 8545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d: Bootstrap starting.
I20260926 02:12:39.682466 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:39.667237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45483 (local address 127.18.159.190:39859)
0926 02:12:39.669571 (+ 2334us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:39.669597 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:39.669654 (+ 57us) server_negotiation.cc:407] Connection header received
0926 02:12:39.672078 (+ 2424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:39.672105 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:39.672685 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:39.673150 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:39.673786 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:39.675118 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:39.676732 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:39.677822 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:39.678086 (+ 264us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:39.679701 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:39.679710 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:39.679728 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:39.680517 (+ 789us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:39.680612 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:39.680641 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:39.680840 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:39.681182 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 02:12:39.681395 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1214,"thread_start_us":902,"threads_started":1}
I20260926 02:12:39.684909 8545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:39.686345 8545 log.cc:824] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:39.687263 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:12:39.690104 8545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d: No bootstrap required, opened a new log
I20260926 02:12:39.700078 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:39.692845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51501 (local address 127.18.159.190:39859)
0926 02:12:39.693053 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:39.693060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:39.693448 (+ 388us) server_negotiation.cc:407] Connection header received
0926 02:12:39.693609 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:39.693614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:39.693712 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:39.693840 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:39.694372 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:39.695555 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:39.697058 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:39.697816 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:39.697955 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:39.699239 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:39.699247 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:39.699252 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:39.699279 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:39.699338 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:39.699369 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:39.699493 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:39.699675 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 02:12:39.699734 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 02:12:39.701066 8545 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c1f95d4a6b426caddc310647fdc64d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39859 } }
I20260926 02:12:39.701738 8545 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:39.701915 8545 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c1f95d4a6b426caddc310647fdc64d, State: Initialized, Role: FOLLOWER
I20260926 02:12:39.702529 8545 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c1f95d4a6b426caddc310647fdc64d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39859 } }
I20260926 02:12:39.702788 8545 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:12:39.702956 8545 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:12:39.703140 8545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:39.705566 8545 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c1f95d4a6b426caddc310647fdc64d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39859 } }
I20260926 02:12:39.706177 8545 leader_election.cc:302] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8c1f95d4a6b426caddc310647fdc64d; no voters:
I20260926 02:12:39.706938 8545 leader_election.cc:288] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:12:39.707087 8555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:39.707890 8555 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d [term 1 LEADER]: Becoming Leader. State: Replica: a8c1f95d4a6b426caddc310647fdc64d, State: Running, Role: LEADER
I20260926 02:12:39.708496 8555 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c1f95d4a6b426caddc310647fdc64d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39859 } }
I20260926 02:12:39.709231 8545 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:39.711536 8557 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8c1f95d4a6b426caddc310647fdc64d. Latest consensus state: current_term: 1 leader_uuid: "a8c1f95d4a6b426caddc310647fdc64d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c1f95d4a6b426caddc310647fdc64d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39859 } } }
I20260926 02:12:39.712376 8557 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:39.713255 8556 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8c1f95d4a6b426caddc310647fdc64d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c1f95d4a6b426caddc310647fdc64d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 39859 } } }
I20260926 02:12:39.713902 8556 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:39.716001 8563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:39.721146 8563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:39.730930 8563 catalog_manager.cc:1347] Generated new cluster ID: b666dba62d8d41cca4866095eb596fb8
I20260926 02:12:39.731091 8563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:39.767911 8563 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:39.769976 8563 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:39.785137 8563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8c1f95d4a6b426caddc310647fdc64d: Generated new TSK 0
I20260926 02:12:39.786211 8563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:12:39.798471 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 8478
W20260926 02:12:39.799804 8540 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45483
W20260926 02:12:39.800776 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 8478: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:12:39.800961 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 8478
W20260926 02:12:39.819111 8463 connection.cc:535] client connection to 127.18.159.190:39859 recv error: Network error: failed to read from TLS socket (remote: 127.18.159.190:39859): Connection reset by peer (error 104)
2026-09-26T02:12:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1988 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:12:39 dist-test-slave-03rn krb5kdc[8576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:12:39 dist-test-slave-03rn krb5kdc[8576](info): set up 2 sockets
Sep 26 02:12:39 dist-test-slave-03rn krb5kdc[8576](info): commencing operation
krb5kdc: starting...
W20260926 02:12:40.925550 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:12:41 dist-test-slave-03rn krb5kdc[8576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:12:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:12:41.201725 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:44273
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:44273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:41.551061 8589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:41.551409 8589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:41.551571 8589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:41.559005 8589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:41.559154 8589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:41.559293 8589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:41.559396 8589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:41.559465 8589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:41.570066 8589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:44273
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:44273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:41.571481 8589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:41.572882 8589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:41.587064 8597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:41.587008 8595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:41.587039 8594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:41.588189 8589 server_base.cc:1034] running on GCE node
I20260926 02:12:41.589440 8589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:41.591567 8589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:41.592805 8589 hybrid_clock.cc:648] HybridClock initialized: now 1790388761592758 us; error 71 us; skew 500 ppm
I20260926 02:12:41.593281 8589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:41 dist-test-slave-03rn krb5kdc[8576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388761, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:41.602233 8589 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:12:41.603896 8589 webserver.cc:460] Webserver started at http://127.18.159.190:37789/ using document root <none> and password file <none>
I20260926 02:12:41.604743 8589 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:41.604893 8589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:41.605230 8589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:41.608472 8589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "1adafcccd19940199c84fa34778d01e7"
format_stamp: "Formatted at 2026-09-26 02:12:41 on dist-test-slave-03rn"
server_key: "b7cbb5fdd96b0500cd2ad70fec262e71"
server_key_iv: "a650f730c4d72e9e76590cda7664839b"
server_key_version: "encryptionkey@0"
I20260926 02:12:41.609319 8589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "1adafcccd19940199c84fa34778d01e7"
format_stamp: "Formatted at 2026-09-26 02:12:41 on dist-test-slave-03rn"
server_key: "b7cbb5fdd96b0500cd2ad70fec262e71"
server_key_iv: "a650f730c4d72e9e76590cda7664839b"
server_key_version: "encryptionkey@0"
I20260926 02:12:41.615543 8589 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 02:12:41.619378 8604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:41.621009 8589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:12:41.621346 8589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "1adafcccd19940199c84fa34778d01e7"
format_stamp: "Formatted at 2026-09-26 02:12:41 on dist-test-slave-03rn"
server_key: "b7cbb5fdd96b0500cd2ad70fec262e71"
server_key_iv: "a650f730c4d72e9e76590cda7664839b"
server_key_version: "encryptionkey@0"
I20260926 02:12:41.621577 8589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:41.627772 8589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:41.634896 8589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:41.635293 8589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:41.655625 8655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:44273 every 8 connection(s)
I20260926 02:12:41.655582 8589 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:44273
I20260926 02:12:41.657341 8589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:41.659134 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 8589
I20260926 02:12:41.659322 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:41.659912 19070 external_mini_cluster.cc:1428] Setting key 9de19fd7f3412f2ae700fd25c60c045b
I20260926 02:12:41.662321 8656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:41.668550 8659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:41.661933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40671 (local address 127.18.159.190:44273)
0926 02:12:41.662845 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:41.662858 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:41.662903 (+ 45us) server_negotiation.cc:407] Connection header received
0926 02:12:41.664679 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:41.664703 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:41.665130 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:41.665537 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:41.666737 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:41.666744 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:41.666758 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:41.666772 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:41.666861 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:41.666882 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:41.667072 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:41.667179 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 02:12:41.667406 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":447,"threads_started":1}
I20260926 02:12:41.672960 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:12:41.676102 8656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7: Bootstrap starting.
I20260926 02:12:41.680218 8656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:41.681608 8656 log.cc:824] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:41.683148 8659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:41.678663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36937 (local address 127.18.159.190:44273)
0926 02:12:41.678877 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:41.678883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:41.680875 (+ 1992us) server_negotiation.cc:407] Connection header received
0926 02:12:41.681035 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:41.681041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:41.681108 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:41.681234 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:41.682392 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:41.682398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:41.682403 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:41.682430 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:41.682477 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:41.682496 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:41.682664 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:41.682736 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 02:12:41.682806 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:12:41.685153 8656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7: No bootstrap required, opened a new log
I20260926 02:12:41.693602 8656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adafcccd19940199c84fa34778d01e7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44273 } }
I20260926 02:12:41.694062 8656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:41.694199 8656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adafcccd19940199c84fa34778d01e7, State: Initialized, Role: FOLLOWER
I20260926 02:12:41.694936 8656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adafcccd19940199c84fa34778d01e7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44273 } }
I20260926 02:12:41.695432 8656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:12:41.695672 8656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:12:41.695940 8656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:41.698480 8656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adafcccd19940199c84fa34778d01e7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44273 } }
I20260926 02:12:41.699098 8656 leader_election.cc:302] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1adafcccd19940199c84fa34778d01e7; no voters:
I20260926 02:12:41.699743 8656 leader_election.cc:288] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:12:41.699940 8666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:41.700652 8666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1adafcccd19940199c84fa34778d01e7, State: Running, Role: LEADER
I20260926 02:12:41.701310 8666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adafcccd19940199c84fa34778d01e7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44273 } }
I20260926 02:12:41.702104 8656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:41.704821 8668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1adafcccd19940199c84fa34778d01e7. Latest consensus state: current_term: 1 leader_uuid: "1adafcccd19940199c84fa34778d01e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adafcccd19940199c84fa34778d01e7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44273 } } }
I20260926 02:12:41.704864 8667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1adafcccd19940199c84fa34778d01e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adafcccd19940199c84fa34778d01e7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 44273 } } }
I20260926 02:12:41.705261 8668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:41.705272 8667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:41.709266 8672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:41.714661 8672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:41.725196 8672 catalog_manager.cc:1347] Generated new cluster ID: 324bc09bee334e17856a9351171499f9
I20260926 02:12:41.725396 8672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:41.745304 8672 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:41.745862 8672 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:41.754966 8672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1adafcccd19940199c84fa34778d01e7: Generated new TSK 0
I20260926 02:12:41.756145 8672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:12:41.766172 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 8589
W20260926 02:12:41.767477 8651 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40671
W20260926 02:12:41.768270 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 8589: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:12:41.768437 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 8589
2026-09-26T02:12:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1966 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:12:41 dist-test-slave-03rn krb5kdc[8687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:12:41 dist-test-slave-03rn krb5kdc[8687](info): set up 2 sockets
Sep 26 02:12:41 dist-test-slave-03rn krb5kdc[8687](info): commencing operation
krb5kdc: starting...
W20260926 02:12:42.881028 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:12:43 dist-test-slave-03rn krb5kdc[8687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:12:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:12:43.151638 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:43389
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:42099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:43389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:43.501195 8700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:43.501518 8700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:43.501708 8700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:43.509208 8700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:43.509335 8700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:43.509465 8700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:43.509575 8700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:43.509663 8700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:43.520077 8700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:42099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:43389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:43389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:43.521610 8700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:43.523036 8700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:43.535831 8708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:43.535946 8705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:43.535945 8706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:43.536885 8700 server_base.cc:1034] running on GCE node
I20260926 02:12:43.537798 8700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:43.539346 8700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:43.540550 8700 hybrid_clock.cc:648] HybridClock initialized: now 1790388763540508 us; error 35 us; skew 500 ppm
I20260926 02:12:43.540894 8700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:43 dist-test-slave-03rn krb5kdc[8687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388763, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:43.543977 8700 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:12:43.545686 8700 webserver.cc:460] Webserver started at http://127.18.159.190:33541/ using document root <none> and password file <none>
I20260926 02:12:43.546387 8700 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:43.546499 8700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:43.546756 8700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:43.549983 8700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "c8816751a57d4de787087af16436e977"
format_stamp: "Formatted at 2026-09-26 02:12:43 on dist-test-slave-03rn"
server_key: "4d709797d8ac3ee3648b38385cd24b23"
server_key_iv: "0fde081059ebecb79ad5cc24f82bc14c"
server_key_version: "encryptionkey@0"
I20260926 02:12:43.550765 8700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "c8816751a57d4de787087af16436e977"
format_stamp: "Formatted at 2026-09-26 02:12:43 on dist-test-slave-03rn"
server_key: "4d709797d8ac3ee3648b38385cd24b23"
server_key_iv: "0fde081059ebecb79ad5cc24f82bc14c"
server_key_version: "encryptionkey@0"
I20260926 02:12:43.556432 8700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 02:12:43.560424 8715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:43.561996 8700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:12:43.562166 8700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "c8816751a57d4de787087af16436e977"
format_stamp: "Formatted at 2026-09-26 02:12:43 on dist-test-slave-03rn"
server_key: "4d709797d8ac3ee3648b38385cd24b23"
server_key_iv: "0fde081059ebecb79ad5cc24f82bc14c"
server_key_version: "encryptionkey@0"
I20260926 02:12:43.562372 8700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:43.574375 8700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:43.578780 8700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:43.579046 8700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:43.598592 8700 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:43389
I20260926 02:12:43.598611 8766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:43389 every 8 connection(s)
I20260926 02:12:43.600131 8700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:43.604131 8767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:43.608135 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 8700
I20260926 02:12:43.608359 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:43.608886 19070 external_mini_cluster.cc:1428] Setting key 675abdbdf28614c94ea1121276f86109
I20260926 02:12:43.616986 8767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977: Bootstrap starting.
I20260926 02:12:43.620177 8767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:43.621304 8767 log.cc:824] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:12:43 dist-test-slave-03rn krb5kdc[8687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:12:43.624261 8767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977: No bootstrap required, opened a new log
I20260926 02:12:43.631801 8770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:43.610855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52535 (local address 127.18.159.190:43389)
0926 02:12:43.611652 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:43.611664 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:43.611708 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:12:43.613442 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:43.613467 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:43.613923 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:43.614370 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:43.615903 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:43.617449 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:43.618796 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:43.619478 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:43.619673 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:43.623768 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:43.623777 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:43.623792 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:43.623827 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:43.627260 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:43.628078 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:43.628095 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:43.628110 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:43.628361 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:43.628786 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:43.628791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:43.628795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:43.629396 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:43.629720 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:43.630221 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 02:12:43.630714 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":373,"threads_started":1}
I20260926 02:12:43.632571 8767 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8816751a57d4de787087af16436e977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 43389 } }
I20260926 02:12:43.633065 8767 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:43.633191 8767 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8816751a57d4de787087af16436e977, State: Initialized, Role: FOLLOWER
I20260926 02:12:43.633847 8767 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8816751a57d4de787087af16436e977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 43389 } }
I20260926 02:12:43.634128 8767 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:12:43.634285 8767 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:12:43.634497 8767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:43.636274 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:12:43.637238 8767 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8816751a57d4de787087af16436e977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 43389 } }
I20260926 02:12:43.637766 8767 leader_election.cc:302] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8816751a57d4de787087af16436e977; no voters:
I20260926 02:12:43.638365 8767 leader_election.cc:288] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:12:43.638556 8772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:43.640547 8767 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:43.647555 8772 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977 [term 1 LEADER]: Becoming Leader. State: Replica: c8816751a57d4de787087af16436e977, State: Running, Role: LEADER
I20260926 02:12:43.648376 8772 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8816751a57d4de787087af16436e977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 43389 } }
I20260926 02:12:43.653088 8773 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8816751a57d4de787087af16436e977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8816751a57d4de787087af16436e977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 43389 } } }
I20260926 02:12:43.653618 8773 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:43.653067 8775 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8816751a57d4de787087af16436e977. Latest consensus state: current_term: 1 leader_uuid: "c8816751a57d4de787087af16436e977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8816751a57d4de787087af16436e977" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 43389 } } }
I20260926 02:12:43.654495 8775 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977 [sys.catalog]: This master's current role is: LEADER
W20260926 02:12:43.671717 8791 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:12:43.671883 8791 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:12:43.672003 8794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:43.673863 8770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:43.649977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49567 (local address 127.18.159.190:43389)
0926 02:12:43.650188 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:43.650198 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:43.650734 (+ 536us) server_negotiation.cc:407] Connection header received
0926 02:12:43.650898 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:43.650907 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:43.651016 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:43.651146 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:43.654583 (+ 3437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:43.656112 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:43.658743 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:43.659740 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:43.659887 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:43.665071 (+ 5184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:43.665081 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:43.665088 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:43.665153 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:43.669273 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:43.670351 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:43.670361 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:43.670364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:43.670530 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:43.671063 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:43.671067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:43.671070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:43.671590 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:43.673007 (+ 1417us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:43.673175 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 02:12:43.673481 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 02:12:43.674360 8794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:43.685382 8794 catalog_manager.cc:1347] Generated new cluster ID: ba57d8f47d4b4c86b8fc284c3d7ef428
I20260926 02:12:43.685536 8794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:43.703981 8794 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:43.705736 8794 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:43.718881 8794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8816751a57d4de787087af16436e977: Generated new TSK 0
I20260926 02:12:43.719851 8794 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:12:43.731887 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 8700
I20260926 02:12:43.733223 8762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:43.827279 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 8700
2026-09-26T02:12:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2058 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:12:43 dist-test-slave-03rn krb5kdc[8800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:12:43 dist-test-slave-03rn krb5kdc[8800](info): set up 2 sockets
Sep 26 02:12:43 dist-test-slave-03rn krb5kdc[8800](info): commencing operation
krb5kdc: starting...
W20260926 02:12:44.937536 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:12:45 dist-test-slave-03rn krb5kdc[8800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:12:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:12:45.216439 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:43371
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:43371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:45.573450 8813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:45.573765 8813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:45.573859 8813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:45.581414 8813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:45.581547 8813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:45.581594 8813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:45.581656 8813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:45.581696 8813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:45.591990 8813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:43371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:43371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:45.593441 8813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:45.594805 8813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:45.608201 8818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:45.608254 8821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:45.608143 8819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:45.609474 8813 server_base.cc:1034] running on GCE node
I20260926 02:12:45.610409 8813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:45.612412 8813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:45.613669 8813 hybrid_clock.cc:648] HybridClock initialized: now 1790388765613621 us; error 41 us; skew 500 ppm
I20260926 02:12:45.614054 8813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:45 dist-test-slave-03rn krb5kdc[8800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388765, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:45.617498 8813 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:12:45.619113 8813 webserver.cc:460] Webserver started at http://127.18.159.190:42573/ using document root <none> and password file <none>
I20260926 02:12:45.619858 8813 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:45.620010 8813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:45.620390 8813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:45.623544 8813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "a2a8a6c515894f3b87fbc4309d9eea17"
format_stamp: "Formatted at 2026-09-26 02:12:45 on dist-test-slave-03rn"
server_key: "7d8099a456d4d3ca365b362dda4c61b5"
server_key_iv: "42d5b47f96ae7b849785cc9f6151917c"
server_key_version: "encryptionkey@0"
I20260926 02:12:45.624440 8813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "a2a8a6c515894f3b87fbc4309d9eea17"
format_stamp: "Formatted at 2026-09-26 02:12:45 on dist-test-slave-03rn"
server_key: "7d8099a456d4d3ca365b362dda4c61b5"
server_key_iv: "42d5b47f96ae7b849785cc9f6151917c"
server_key_version: "encryptionkey@0"
I20260926 02:12:45.630038 8813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:12:45.634121 8828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:45.635560 8813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:12:45.635776 8813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "a2a8a6c515894f3b87fbc4309d9eea17"
format_stamp: "Formatted at 2026-09-26 02:12:45 on dist-test-slave-03rn"
server_key: "7d8099a456d4d3ca365b362dda4c61b5"
server_key_iv: "42d5b47f96ae7b849785cc9f6151917c"
server_key_version: "encryptionkey@0"
I20260926 02:12:45.635994 8813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:45.648241 8813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:45.653182 8813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:45.653543 8813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:45.673935 8813 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:43371
I20260926 02:12:45.673938 8879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:43371 every 8 connection(s)
I20260926 02:12:45.675501 8813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:45.679657 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:45.684114 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 8813
I20260926 02:12:45.684441 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:45.684895 19070 external_mini_cluster.cc:1428] Setting key 57aab38e7cfef9e01c711c07f0664b9f
I20260926 02:12:45.692337 8880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17: Bootstrap starting.
I20260926 02:12:45.696668 8880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:45.698165 8880 log.cc:824] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:45.702066 8883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:45.686982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34255 (local address 127.18.159.190:43371)
0926 02:12:45.687849 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:45.687863 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:45.687909 (+ 46us) server_negotiation.cc:407] Connection header received
0926 02:12:45.690203 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:45.690236 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:45.690835 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:45.691488 (+ 653us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:45.692623 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:45.694797 (+ 2174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:45.696370 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:45.697569 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:45.697807 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:45.699290 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:45.699302 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:45.699330 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:45.700126 (+ 796us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:45.700289 (+ 163us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:45.700326 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:45.700478 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:45.700696 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 02:12:45.701005 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":394,"threads_started":1}
I20260926 02:12:45.702569 8880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17: No bootstrap required, opened a new log
I20260926 02:12:45.706627 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:12:45.711997 8880 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a8a6c515894f3b87fbc4309d9eea17" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 43371 } }
I20260926 02:12:45.712576 8880 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:45.712700 8880 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a8a6c515894f3b87fbc4309d9eea17, State: Initialized, Role: FOLLOWER
I20260926 02:12:45.713418 8880 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a8a6c515894f3b87fbc4309d9eea17" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 43371 } }
I20260926 02:12:45.713697 8880 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:12:45.713874 8880 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:12:45.714071 8880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:45.716542 8880 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a8a6c515894f3b87fbc4309d9eea17" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 43371 } }
I20260926 02:12:45.717126 8880 leader_election.cc:302] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2a8a6c515894f3b87fbc4309d9eea17; no voters:
I20260926 02:12:45.717871 8880 leader_election.cc:288] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:12:45.718006 8890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:45.718267 8883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:45.711234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38277 (local address 127.18.159.190:43371)
0926 02:12:45.711498 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:45.711506 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:45.711922 (+ 416us) server_negotiation.cc:407] Connection header received
0926 02:12:45.712282 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:45.712287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:45.712363 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:45.712458 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:45.713165 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:45.714122 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:45.715506 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:45.716006 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:45.716158 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:45.717386 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:45.717392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:45.717397 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:45.717427 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:45.717488 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:45.717507 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:45.717617 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:45.717766 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 02:12:45.717875 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:12:45.719362 8890 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17 [term 1 LEADER]: Becoming Leader. State: Replica: a2a8a6c515894f3b87fbc4309d9eea17, State: Running, Role: LEADER
I20260926 02:12:45.720057 8890 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a8a6c515894f3b87fbc4309d9eea17" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 43371 } }
I20260926 02:12:45.720686 8880 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:45.723308 8891 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2a8a6c515894f3b87fbc4309d9eea17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a8a6c515894f3b87fbc4309d9eea17" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 43371 } } }
I20260926 02:12:45.723909 8891 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:45.723731 8892 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2a8a6c515894f3b87fbc4309d9eea17. Latest consensus state: current_term: 1 leader_uuid: "a2a8a6c515894f3b87fbc4309d9eea17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a8a6c515894f3b87fbc4309d9eea17" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 43371 } } }
I20260926 02:12:45.724100 8892 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:45.727150 8896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:45.732488 8896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:45.741919 8896 catalog_manager.cc:1347] Generated new cluster ID: 8d4d7100875e46a29241cf60b8e966b1
I20260926 02:12:45.742062 8896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:45.762826 8896 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:45.764303 8896 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:45.770555 8896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2a8a6c515894f3b87fbc4309d9eea17: Generated new TSK 0
I20260926 02:12:45.771493 8896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:12:45.786927 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 8813
W20260926 02:12:45.788066 8875 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34255
W20260926 02:12:45.788861 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 8813: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:12:45.788985 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 8813
2026-09-26T02:12:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1961 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:12:45 dist-test-slave-03rn krb5kdc[8911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:12:45 dist-test-slave-03rn krb5kdc[8911](info): set up 2 sockets
Sep 26 02:12:45 dist-test-slave-03rn krb5kdc[8911](info): commencing operation
krb5kdc: starting...
W20260926 02:12:46.917513 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:12:47 dist-test-slave-03rn krb5kdc[8911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:12:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:12:47.189855 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:32777
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:44103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:32777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:47.534116 8924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:47.534430 8924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:47.534603 8924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:47.542207 8924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:47.542342 8924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:47.542441 8924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:47.542552 8924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:47.542634 8924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:47.552804 8924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:44103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:32777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:32777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:47.554181 8924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:47.555528 8924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:47.567705 8929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:47.567723 8930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:47.567839 8932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:47.568991 8924 server_base.cc:1034] running on GCE node
I20260926 02:12:47.569877 8924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:47.571357 8924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:47.572609 8924 hybrid_clock.cc:648] HybridClock initialized: now 1790388767572575 us; error 41 us; skew 500 ppm
I20260926 02:12:47.573027 8924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:47 dist-test-slave-03rn krb5kdc[8911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388767, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:47.576215 8924 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:12:47.577802 8924 webserver.cc:460] Webserver started at http://127.18.159.190:38345/ using document root <none> and password file <none>
I20260926 02:12:47.578524 8924 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:47.578670 8924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:47.578990 8924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:47.582134 8924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "2f4ae32e6ae64b2cabaa536c34018942"
format_stamp: "Formatted at 2026-09-26 02:12:47 on dist-test-slave-03rn"
server_key: "adfa3b05c0af1502856b602b55d1f501"
server_key_iv: "1dd3c1cb0621b0e09fb83f3abe59ce11"
server_key_version: "encryptionkey@0"
I20260926 02:12:47.582962 8924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "2f4ae32e6ae64b2cabaa536c34018942"
format_stamp: "Formatted at 2026-09-26 02:12:47 on dist-test-slave-03rn"
server_key: "adfa3b05c0af1502856b602b55d1f501"
server_key_iv: "1dd3c1cb0621b0e09fb83f3abe59ce11"
server_key_version: "encryptionkey@0"
I20260926 02:12:47.588438 8924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260926 02:12:47.592113 8939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:47.593701 8924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:12:47.593902 8924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "2f4ae32e6ae64b2cabaa536c34018942"
format_stamp: "Formatted at 2026-09-26 02:12:47 on dist-test-slave-03rn"
server_key: "adfa3b05c0af1502856b602b55d1f501"
server_key_iv: "1dd3c1cb0621b0e09fb83f3abe59ce11"
server_key_version: "encryptionkey@0"
I20260926 02:12:47.594122 8924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:47.600346 8924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:47.604811 8924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:47.605146 8924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:47.624514 8924 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:32777
I20260926 02:12:47.624513 8990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:32777 every 8 connection(s)
I20260926 02:12:47.626091 8924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:47.630292 8991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:47.636514 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 8924
I20260926 02:12:47.636721 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:47.637138 19070 external_mini_cluster.cc:1428] Setting key 87d0112fea853f28af414a017ffbdf2b
I20260926 02:12:47.642324 8991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942: Bootstrap starting.
I20260926 02:12:47.645483 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:47.639033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35163 (local address 127.18.159.190:32777)
0926 02:12:47.639965 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:47.639979 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:47.640025 (+ 46us) server_negotiation.cc:407] Connection header received
0926 02:12:47.641815 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:47.641841 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:47.642213 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:47.642633 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:47.643738 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:47.643745 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:47.643766 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:47.643785 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:47.643844 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:47.643864 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:47.644114 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:47.644298 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 02:12:47.644508 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":407,"threads_started":1}
I20260926 02:12:47.646109 8991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:47.647271 8991 log.cc:824] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:47.649674 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:12:47.650718 8991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942: No bootstrap required, opened a new log
I20260926 02:12:47.657024 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:47.654348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35277 (local address 127.18.159.190:32777)
0926 02:12:47.654539 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:47.654546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:47.654885 (+ 339us) server_negotiation.cc:407] Connection header received
0926 02:12:47.655026 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:47.655032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:47.655101 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:47.655231 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:47.656327 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:47.656334 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:47.656338 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:12:47.656357 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:47.656413 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:12:47.656434 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:47.656571 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:47.656654 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 02:12:47.656719 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:12:47.659749 8991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f4ae32e6ae64b2cabaa536c34018942" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32777 } }
I20260926 02:12:47.660251 8991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:12:47.660347 8991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4ae32e6ae64b2cabaa536c34018942, State: Initialized, Role: FOLLOWER
I20260926 02:12:47.660895 8991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f4ae32e6ae64b2cabaa536c34018942" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32777 } }
I20260926 02:12:47.661128 8991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:12:47.661232 8991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:12:47.661366 8991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:47.663710 8991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f4ae32e6ae64b2cabaa536c34018942" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32777 } }
I20260926 02:12:47.664392 8991 leader_election.cc:302] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f4ae32e6ae64b2cabaa536c34018942; no voters:
I20260926 02:12:47.665226 8991 leader_election.cc:288] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:12:47.665292 9001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:47.666072 9001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942 [term 1 LEADER]: Becoming Leader. State: Replica: 2f4ae32e6ae64b2cabaa536c34018942, State: Running, Role: LEADER
I20260926 02:12:47.666713 9001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f4ae32e6ae64b2cabaa536c34018942" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32777 } }
I20260926 02:12:47.667486 8991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:47.669885 9002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f4ae32e6ae64b2cabaa536c34018942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f4ae32e6ae64b2cabaa536c34018942" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32777 } } }
I20260926 02:12:47.669821 9003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f4ae32e6ae64b2cabaa536c34018942. Latest consensus state: current_term: 1 leader_uuid: "2f4ae32e6ae64b2cabaa536c34018942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f4ae32e6ae64b2cabaa536c34018942" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 32777 } } }
I20260926 02:12:47.670356 9002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:47.670424 9003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:47.671290 9008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:47.678031 9008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:47.688374 9008 catalog_manager.cc:1347] Generated new cluster ID: 93f8c7404772405aadbd637b57821827
I20260926 02:12:47.688526 9008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:47.706769 9008 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:47.707329 9008 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:47.716598 9008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f4ae32e6ae64b2cabaa536c34018942: Generated new TSK 0
I20260926 02:12:47.717556 9008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:12:47.731504 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 8924
W20260926 02:12:47.732787 8986 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35163
W20260926 02:12:47.733562 19070 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 8924: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:12:47.733716 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 8924
2026-09-26T02:12:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1943 ms)
[----------] 18 tests from AuthTokenIssuingTest (36139 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:12:47 dist-test-slave-03rn krb5kdc[9022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:12:47 dist-test-slave-03rn krb5kdc[9022](info): set up 2 sockets
Sep 26 02:12:47 dist-test-slave-03rn krb5kdc[9022](info): commencing operation
krb5kdc: starting...
W20260926 02:12:48.849184 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:12:49 dist-test-slave-03rn krb5kdc[9022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:12:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:12:49.128347 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:36725
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:36725,127.18.159.189:34883,127.18.159.188:39509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:49.478632 9035 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:12:49.478998 9035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:49.479094 9035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:49.479188 9035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:49.486660 9035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:49.486771 9035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:49.486833 9035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:49.486881 9035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:49.486945 9035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:49.497118 9035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:36725,127.18.159.189:34883,127.18.159.188:39509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:36725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:49.498416 9035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:49.499742 9035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:49.512569 9041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:49.512569 9043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:49.512569 9040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:49.517901 9035 server_base.cc:1034] running on GCE node
I20260926 02:12:49.518890 9035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:49.520684 9035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:49.521965 9035 hybrid_clock.cc:648] HybridClock initialized: now 1790388769521876 us; error 76 us; skew 500 ppm
I20260926 02:12:49.522353 9035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:49 dist-test-slave-03rn krb5kdc[9022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388769, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:49.526135 9035 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:12:49.527976 9035 webserver.cc:460] Webserver started at http://127.18.159.190:35703/ using document root <none> and password file <none>
I20260926 02:12:49.528952 9035 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:49.529125 9035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:49.529495 9035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:49.533253 9035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "38b26fd932eb477996c55ba41b9c4d8d"
format_stamp: "Formatted at 2026-09-26 02:12:49 on dist-test-slave-03rn"
server_key: "02ae9cf3711d09a8ca59fb581a90726c"
server_key_iv: "d2613b758ce213fe81dc9c9ada501e97"
server_key_version: "encryptionkey@0"
I20260926 02:12:49.534175 9035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "38b26fd932eb477996c55ba41b9c4d8d"
format_stamp: "Formatted at 2026-09-26 02:12:49 on dist-test-slave-03rn"
server_key: "02ae9cf3711d09a8ca59fb581a90726c"
server_key_iv: "d2613b758ce213fe81dc9c9ada501e97"
server_key_version: "encryptionkey@0"
I20260926 02:12:49.540012 9035 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:12:49.544122 9050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:49.545850 9035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 02:12:49.546103 9035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "38b26fd932eb477996c55ba41b9c4d8d"
format_stamp: "Formatted at 2026-09-26 02:12:49 on dist-test-slave-03rn"
server_key: "02ae9cf3711d09a8ca59fb581a90726c"
server_key_iv: "d2613b758ce213fe81dc9c9ada501e97"
server_key_version: "encryptionkey@0"
I20260926 02:12:49.546376 9035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:49.569577 9035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:49.574612 9035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:49.574934 9035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:49.595824 9035 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:36725
I20260926 02:12:49.595873 9101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:36725 every 8 connection(s)
I20260926 02:12:49.597308 9035 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:49.601466 9102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:49.606185 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 9035
I20260926 02:12:49.606448 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:49.607067 19070 external_mini_cluster.cc:1428] Setting key 2884b6d95b372382e073d17230ba5846
I20260926 02:12:49.608803 9102 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36725 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:12:49 dist-test-slave-03rn krb5kdc[9022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388769, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.159.189@KRBTEST.COM; defaulting to no policy
I20260926 02:12:49.633004 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:49.612117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:36725 (local address 127.0.0.1:50428)
0926 02:12:49.613099 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:49.613156 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:49.615307 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:49.616295 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:49.616302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:49.617218 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:49.617638 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:49.617668 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:49.619333 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:49.619338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:49.620539 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:49.620551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:49.621526 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:49.621533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:49.621727 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:49.622450 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:49.622477 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:49.624798 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:49.628441 (+ 3643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:49.628453 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:49.628467 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:49.628894 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:49.629333 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:49.629340 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:49.629343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:49.629505 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:49.630298 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:49.630307 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:49.630541 (+ 234us) client_negotiation.cc:769] Sending connection context
0926 02:12:49.630809 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 02:12:49.631556 (+ 747us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":251,"thread_start_us":411,"threads_started":1}
I20260926 02:12:49.633024 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:49.612120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50428 (local address 127.18.159.190:36725)
0926 02:12:49.613987 (+ 1867us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:49.613997 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:49.614053 (+ 56us) server_negotiation.cc:407] Connection header received
0926 02:12:49.615749 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:49.615755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:49.616005 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:49.616286 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:49.617881 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:49.619166 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:49.620694 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:49.621288 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:49.621504 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:49.625056 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:49.625070 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:49.625083 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:49.625118 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:49.628285 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:49.629051 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:49.629058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:49.629060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:49.629218 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:49.629619 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:49.629626 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:49.629629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:49.630159 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:49.630378 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:49.630942 (+ 564us) server_negotiation.cc:299] Negotiation successful
0926 02:12:49.631431 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":1647,"thread_start_us":1334,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.189@KRBTEST.COM" created.
I20260926 02:12:49.636463 9102 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:12:49.637681 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:49.637242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:34883 (local address 127.0.0.1:38974)
0926 02:12:49.637431 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:49.637551 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:34883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 02:12:49.637959 9053 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.159.189:34883: connect: Connection refused (error 111)
I20260926 02:12:49.639588 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:49.639226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:34883 (local address 127.0.0.1:38984)
0926 02:12:49.639382 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:49.639448 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:34883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 02:12:49.639793 9102 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.189:34883: Network error: Client connection negotiation failed: client connection to 127.18.159.189:34883: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.189.keytab.
Entry for principal kudu/127.18.159.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.189.keytab.
I20260926 02:12:49.667629 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.189:34883
--webserver_interface=127.18.159.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.189.keytab
--principal=kudu/127.18.159.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:36725,127.18.159.189:34883,127.18.159.188:39509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:12:49.689200 9102 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } attempt: 1
I20260926 02:12:49.690442 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:49.689968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:34883 (local address 127.0.0.1:39000)
0926 02:12:49.690176 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:49.690308 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:34883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 02:12:49.691694 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:49.691287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:34883 (local address 127.0.0.1:39004)
0926 02:12:49.691469 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:49.691543 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:34883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 02:12:49.691946 9102 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.189:34883: Network error: Client connection negotiation failed: client connection to 127.18.159.189:34883: connect: Connection refused (error 111)
I20260926 02:12:49.760315 9102 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } attempt: 2
I20260926 02:12:49.761495 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:49.761001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:34883 (local address 127.0.0.1:39012)
0926 02:12:49.761269 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:49.761357 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:34883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 02:12:49.762712 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:49.762338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:34883 (local address 127.0.0.1:39026)
0926 02:12:49.762496 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:49.762573 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:34883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 02:12:49.763033 9102 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.189:34883: Network error: Client connection negotiation failed: client connection to 127.18.159.189:34883: connect: Connection refused (error 111)
I20260926 02:12:49.854349 9102 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } attempt: 3
I20260926 02:12:49.855558 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:49.855110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:34883 (local address 127.0.0.1:39034)
0926 02:12:49.855317 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:49.855406 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:34883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 02:12:49.856670 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:49.856328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:34883 (local address 127.0.0.1:39038)
0926 02:12:49.856460 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:49.856532 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:34883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 02:12:49.856884 9102 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.189:34883: Network error: Client connection negotiation failed: client connection to 127.18.159.189:34883: connect: Connection refused (error 111)
I20260926 02:12:50.000290 9102 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } attempt: 4
I20260926 02:12:50.001750 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.001048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:34883 (local address 127.0.0.1:39054)
0926 02:12:50.001500 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.001600 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:34883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306}
I20260926 02:12:50.003001 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.002565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:34883 (local address 127.0.0.1:39056)
0926 02:12:50.002761 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.002858 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:34883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260926 02:12:50.003206 9102 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.189:34883: Network error: Client connection negotiation failed: client connection to 127.18.159.189:34883: connect: Connection refused (error 111)
W20260926 02:12:50.017758 9109 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:12:50.018213 9109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:50.018352 9109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:50.018496 9109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:50.026290 9109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:50.026453 9109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:50.026573 9109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:50.026674 9109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:50.026772 9109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:50.037358 9109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:36725,127.18.159.189:34883,127.18.159.188:39509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.189.keytab
--principal=kudu/127.18.159.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.189:34883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.159.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:50.038833 9109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:50.040294 9109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:50.054127 9117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:50.054219 9114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:50.054906 9115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:50.055011 9109 server_base.cc:1034] running on GCE node
I20260926 02:12:50.056082 9109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:50.057785 9109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:50.059020 9109 hybrid_clock.cc:648] HybridClock initialized: now 1790388770058953 us; error 43 us; skew 500 ppm
I20260926 02:12:50.059437 9109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:50 dist-test-slave-03rn krb5kdc[9022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388770, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:50.063014 9109 init.cc:377] Logged in from keytab as kudu/127.18.159.189@KRBTEST.COM (short username kudu)
I20260926 02:12:50.064756 9109 webserver.cc:460] Webserver started at http://127.18.159.189:46753/ using document root <none> and password file <none>
I20260926 02:12:50.065527 9109 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:50.065670 9109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:50.066020 9109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:50.069391 9109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-1/data/instance:
uuid: "118d2ea9f5c345a08ad712288f07b3fc"
format_stamp: "Formatted at 2026-09-26 02:12:50 on dist-test-slave-03rn"
server_key: "11403ec963fd6442860bf1cd8be33ec3"
server_key_iv: "476f793a306bb591530955c75affd3a0"
server_key_version: "encryptionkey@0"
I20260926 02:12:50.070336 9109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-1/wal/instance:
uuid: "118d2ea9f5c345a08ad712288f07b3fc"
format_stamp: "Formatted at 2026-09-26 02:12:50 on dist-test-slave-03rn"
server_key: "11403ec963fd6442860bf1cd8be33ec3"
server_key_iv: "476f793a306bb591530955c75affd3a0"
server_key_version: "encryptionkey@0"
I20260926 02:12:50.076221 9109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:12:50.080227 9124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:50.081859 9109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:12:50.082104 9109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-1/wal
uuid: "118d2ea9f5c345a08ad712288f07b3fc"
format_stamp: "Formatted at 2026-09-26 02:12:50 on dist-test-slave-03rn"
server_key: "11403ec963fd6442860bf1cd8be33ec3"
server_key_iv: "476f793a306bb591530955c75affd3a0"
server_key_version: "encryptionkey@0"
I20260926 02:12:50.082372 9109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:50.098804 9109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:50.104005 9109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:50.104727 9109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:50.126318 9109 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.189:34883
I20260926 02:12:50.126338 9175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.189:34883 every 8 connection(s)
I20260926 02:12:50.127774 9109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-1/data/info.pb
I20260926 02:12:50.131975 9176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:50.135272 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 9109
I20260926 02:12:50.135475 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-1/wal/instance
I20260926 02:12:50.136152 19070 external_mini_cluster.cc:1428] Setting key 3b6a14e349d74e68ac21dbe7a1c914e9
I20260926 02:12:50.142283 9176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36725 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:12:50 dist-test-slave-03rn krb5kdc[9022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388770, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.189@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.159.188@KRBTEST.COM; defaulting to no policy
I20260926 02:12:50.168843 9178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.146170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50430 (local address 127.18.159.190:36725)
0926 02:12:50.146805 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:50.146811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:50.147958 (+ 1147us) server_negotiation.cc:407] Connection header received
0926 02:12:50.150280 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:50.150285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:50.150371 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:50.150481 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:50.152947 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.153692 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.156361 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.156810 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.156921 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.161776 (+ 4855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:50.161783 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:50.161786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:50.161835 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:50.165090 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.166074 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.166080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.166082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.166258 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.166774 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.166784 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.166805 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.167126 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:50.167304 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:50.168301 (+ 997us) server_negotiation.cc:299] Negotiation successful
0926 02:12:50.168514 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":299,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.188@KRBTEST.COM" created.
I20260926 02:12:50.169956 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.146813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:36725 (local address 127.0.0.1:50430)
0926 02:12:50.147725 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.147788 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:50.149836 (+ 2048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:50.150800 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:50.150816 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:50.151694 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:50.152664 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:50.152698 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.153901 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.153909 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:50.156096 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:50.156125 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.157088 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.157097 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:50.157442 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.158315 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:50.158357 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:50.161439 (+ 3082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:50.165313 (+ 3874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:50.165324 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:50.165345 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:50.165896 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:50.166395 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:50.166401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:50.166405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:50.166623 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:50.167282 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:50.167291 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:50.167743 (+ 452us) client_negotiation.cc:769] Sending connection context
0926 02:12:50.168071 (+ 328us) client_negotiation.cc:240] Negotiation successful
0926 02:12:50.168589 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":369,"threads_started":1}
I20260926 02:12:50.172000 9176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:12:50 dist-test-slave-03rn krb5kdc[9022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388770, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.189@KRBTEST.COM for kudu/127.18.159.189@KRBTEST.COM
I20260926 02:12:50.190119 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.172931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:34883 (local address 127.0.0.1:39070)
0926 02:12:50.173144 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.173165 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:50.173292 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:50.174371 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:50.174379 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:50.174926 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:50.175193 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:50.175209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.176734 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.176742 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:50.178298 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:50.178319 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.179343 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.179351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:50.179542 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.180348 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:50.180366 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:50.182764 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:50.187516 (+ 4752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:50.187520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:50.187524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:50.187917 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:50.188430 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:50.188434 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:50.188436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:50.188579 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:50.189389 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:50.189392 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:50.189511 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 02:12:50.189620 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 02:12:50.189844 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":1}
I20260926 02:12:50.190651 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.173139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39070 (local address 127.18.159.189:34883)
0926 02:12:50.173883 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:50.173890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:50.173918 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:12:50.173966 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:50.173972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:50.174183 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:50.174306 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:50.175413 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.176582 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.178489 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.179156 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.179284 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.183044 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:50.183064 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:50.183093 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:50.183139 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:50.187353 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.188083 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.188096 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.188102 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.188305 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.188696 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.188704 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.188710 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.189258 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:50.189455 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:50.189821 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 02:12:50.190142 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":572,"thread_start_us":417,"threads_started":1}
I20260926 02:12:50.193199 9176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:12:50.194416 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.193941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:39509 (local address 127.0.0.1:51742)
0926 02:12:50.194135 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.194239 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 02:12:50.194700 9125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
I20260926 02:12:50.196451 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.195986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:39509 (local address 127.0.0.1:51752)
0926 02:12:50.196218 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.196341 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 02:12:50.196694 9176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:39509: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.188.keytab.
Entry for principal kudu/127.18.159.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.188.keytab.
I20260926 02:12:50.208045 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.188:39509
--webserver_interface=127.18.159.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.188.keytab
--principal=kudu/127.18.159.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:36725,127.18.159.189:34883,127.18.159.188:39509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:12:50.246029 9176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } attempt: 1
I20260926 02:12:50.247462 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.246847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:39509 (local address 127.0.0.1:51764)
0926 02:12:50.247097 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.247252 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 02:12:50.248735 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.248288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:39509 (local address 127.0.0.1:51774)
0926 02:12:50.248474 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.248576 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 02:12:50.249056 9176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:39509: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
I20260926 02:12:50.302495 9102 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } attempt: 5
Sep 26 02:12:50 dist-test-slave-03rn krb5kdc[9022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388769, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for kudu/127.18.159.189@KRBTEST.COM
I20260926 02:12:50.317353 9176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } attempt: 2
I20260926 02:12:50.318279 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.303263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:34883 (local address 127.0.0.1:39080)
0926 02:12:50.303544 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.303568 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:50.303720 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:50.304089 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:50.304095 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:50.304640 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:50.304895 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:50.304909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.306013 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.306021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:50.307526 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:50.307546 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.308430 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.308436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:50.308665 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.309404 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:50.309428 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:50.312007 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:50.315317 (+ 3310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:50.315325 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:50.315331 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:50.315950 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:50.316507 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:50.316512 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:50.316516 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:50.316701 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:50.317262 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:50.317266 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:50.317437 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 02:12:50.317589 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 02:12:50.317944 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 02:12:50.318427 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.303446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.18.159.189:34883)
0926 02:12:50.303619 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:50.303623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:50.303645 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:12:50.303856 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:50.303860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:50.303925 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:50.304013 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:50.305083 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.305838 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.307703 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.308123 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.308401 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.312332 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:50.312338 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:50.312341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:50.312388 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:50.315139 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.316117 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.316121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.316124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.316376 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.316822 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.316825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.316828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.317125 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:50.317245 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:50.317845 (+ 600us) server_negotiation.cc:299] Negotiation successful
0926 02:12:50.318091 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 02:12:50.318431 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.317955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:39509 (local address 127.0.0.1:51776)
0926 02:12:50.318222 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.318310 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260926 02:12:50.319460 9102 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:12:50.319976 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.319521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:39509 (local address 127.0.0.1:51790)
0926 02:12:50.319734 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.319811 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260926 02:12:50.320258 9176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:39509: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
I20260926 02:12:50.320518 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.320081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:39509 (local address 127.0.0.1:51802)
0926 02:12:50.320274 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.320364 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 02:12:50.321424 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.321081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:39509 (local address 127.0.0.1:51818)
0926 02:12:50.321247 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.321319 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 02:12:50.321612 9102 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:39509: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
I20260926 02:12:50.372848 9102 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } attempt: 1
I20260926 02:12:50.373993 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.373502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:39509 (local address 127.0.0.1:51834)
0926 02:12:50.373715 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.373819 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 02:12:50.375295 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.374826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:39509 (local address 127.0.0.1:51848)
0926 02:12:50.375028 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.375152 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260926 02:12:50.375578 9102 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:39509: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
I20260926 02:12:50.411479 9176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } attempt: 3
I20260926 02:12:50.412741 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.412277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:39509 (local address 127.0.0.1:51858)
0926 02:12:50.412470 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.412584 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 02:12:50.413866 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.413456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:39509 (local address 127.0.0.1:51866)
0926 02:12:50.413679 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.413759 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260926 02:12:50.414072 9176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:39509: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
I20260926 02:12:50.443820 9102 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } attempt: 2
I20260926 02:12:50.445036 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.444536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:39509 (local address 127.0.0.1:51880)
0926 02:12:50.444780 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.444870 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 02:12:50.446130 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.445771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:39509 (local address 127.0.0.1:51896)
0926 02:12:50.445939 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.446012 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 02:12:50.446332 9102 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:39509: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
I20260926 02:12:50.552542 9102 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } attempt: 3
I20260926 02:12:50.553676 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.553237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:39509 (local address 127.0.0.1:51910)
0926 02:12:50.553427 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.553514 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 02:12:50.554798 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.554388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:39509 (local address 127.0.0.1:51916)
0926 02:12:50.554588 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.554672 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260926 02:12:50.555001 9102 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:39509: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
I20260926 02:12:50.557300 9176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } attempt: 4
I20260926 02:12:50.558430 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.557957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:39509 (local address 127.0.0.1:51924)
0926 02:12:50.558189 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.558288 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 02:12:50.559433 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.559095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:39509 (local address 127.0.0.1:51928)
0926 02:12:50.559258 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.559330 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 02:12:50.559636 9176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:39509: Network error: Client connection negotiation failed: client connection to 127.18.159.188:39509: connect: Connection refused (error 111)
W20260926 02:12:50.571058 9184 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:12:50.571487 9184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:50.571611 9184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:50.571746 9184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:50.579684 9184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:50.579814 9184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:50.579938 9184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:50.580022 9184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:50.580134 9184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:50.590540 9184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:36725,127.18.159.189:34883,127.18.159.188:39509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.188.keytab
--principal=kudu/127.18.159.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.188:39509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.159.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:50.591917 9184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:50.593372 9184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:50.607831 9192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:50.607795 9190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:50.607888 9189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:50.608844 9184 server_base.cc:1034] running on GCE node
I20260926 02:12:50.609689 9184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:50.611250 9184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:50.612519 9184 hybrid_clock.cc:648] HybridClock initialized: now 1790388770612440 us; error 53 us; skew 500 ppm
I20260926 02:12:50.612903 9184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:50 dist-test-slave-03rn krb5kdc[9022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388770, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:50.616102 9184 init.cc:377] Logged in from keytab as kudu/127.18.159.188@KRBTEST.COM (short username kudu)
I20260926 02:12:50.617682 9184 webserver.cc:460] Webserver started at http://127.18.159.188:46295/ using document root <none> and password file <none>
I20260926 02:12:50.618423 9184 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:50.618566 9184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:50.618896 9184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:50.622051 9184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-2/data/instance:
uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6"
format_stamp: "Formatted at 2026-09-26 02:12:50 on dist-test-slave-03rn"
server_key: "38e5a44b0026f98f87cb5a0ed09d77ab"
server_key_iv: "f750b13d728566b33b558da58bb7419b"
server_key_version: "encryptionkey@0"
I20260926 02:12:50.622884 9184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-2/wal/instance:
uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6"
format_stamp: "Formatted at 2026-09-26 02:12:50 on dist-test-slave-03rn"
server_key: "38e5a44b0026f98f87cb5a0ed09d77ab"
server_key_iv: "f750b13d728566b33b558da58bb7419b"
server_key_version: "encryptionkey@0"
I20260926 02:12:50.628393 9184 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:12:50.632086 9199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:50.633559 9184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:12:50.633780 9184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-2/wal
uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6"
format_stamp: "Formatted at 2026-09-26 02:12:50 on dist-test-slave-03rn"
server_key: "38e5a44b0026f98f87cb5a0ed09d77ab"
server_key_iv: "f750b13d728566b33b558da58bb7419b"
server_key_version: "encryptionkey@0"
I20260926 02:12:50.634032 9184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:50.646937 9184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:50.651649 9184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:50.651954 9184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:50.673799 9184 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.188:39509
I20260926 02:12:50.673823 9250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.188:39509 every 8 connection(s)
I20260926 02:12:50.675463 9184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-2/data/info.pb
I20260926 02:12:50.679533 9251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:50.685775 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 9184
I20260926 02:12:50.685983 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388402166281-19070-0/minicluster-data/master-2/wal/instance
I20260926 02:12:50.686474 19070 external_mini_cluster.cc:1428] Setting key 12cf8e612a0cd3a5ade17024fab75d81
I20260926 02:12:50.686574 9251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36725 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:12:50 dist-test-slave-03rn krb5kdc[9022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388770, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.188@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
Sep 26 02:12:50 dist-test-slave-03rn krb5kdc[9022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.188@KRBTEST.COM
I20260926 02:12:50.708083 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.689780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.18.159.190:36725)
0926 02:12:50.690347 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:50.690356 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:50.690895 (+ 539us) server_negotiation.cc:407] Connection header received
0926 02:12:50.692117 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:50.692124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:50.692284 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:50.692405 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:50.693752 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.694506 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.696068 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.696572 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.696713 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.700658 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:50.700664 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:50.700667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:50.700703 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:50.704190 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.705178 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.705184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.705186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.705332 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.705908 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.705914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.705917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.706327 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:50.706449 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:50.707359 (+ 910us) server_negotiation.cc:299] Negotiation successful
0926 02:12:50.707638 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":293,"threads_started":1}
I20260926 02:12:50.709313 9254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.688440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.18.159.188:39509)
0926 02:12:50.689321 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:50.689335 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:50.689383 (+ 48us) server_negotiation.cc:407] Connection header received
0926 02:12:50.691807 (+ 2424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:50.691830 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:50.692098 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:50.692379 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:50.693914 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.695130 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.696464 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.697059 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.697196 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.701206 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:50.701220 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:50.701239 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:50.701284 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:50.704932 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.705926 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.705936 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.705939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.706085 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.706640 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.706644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.706646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.707110 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:50.707451 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:50.707782 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 02:12:50.708507 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":609,"spinlock_wait_cycles":17280,"thread_start_us":367,"threads_started":1}
I20260926 02:12:50.709350 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.689729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:36725 (local address 127.0.0.1:50434)
0926 02:12:50.690724 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.690778 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:50.691825 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:50.692487 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:50.692496 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:50.693191 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:50.693568 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:50.693588 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.694673 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.694678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:50.695910 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:50.695921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.696827 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.696834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:50.697105 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.697886 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:50.697922 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:50.700422 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:50.704371 (+ 3949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:50.704390 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:50.704400 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:50.705030 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:50.705477 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:50.705483 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:50.705487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:50.705744 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:50.706495 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:50.706504 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:50.706928 (+ 424us) client_negotiation.cc:769] Sending connection context
0926 02:12:50.707237 (+ 309us) client_negotiation.cc:240] Negotiation successful
0926 02:12:50.707842 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":51,"thread_start_us":645,"threads_started":1}
I20260926 02:12:50.711382 9251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:12:50 dist-test-slave-03rn krb5kdc[9022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388770, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.188@KRBTEST.COM for kudu/127.18.159.189@KRBTEST.COM
Sep 26 02:12:50 dist-test-slave-03rn krb5kdc[9022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:12:50.728570 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.712463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:34883 (local address 127.0.0.1:39096)
0926 02:12:50.712762 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.712782 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:50.712906 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:50.714717 (+ 1811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:50.714720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:50.715069 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:50.715207 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:50.715215 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.716210 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.716215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:50.717114 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:50.717125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.717848 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.717854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:50.718066 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.719206 (+ 1140us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:50.719225 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:50.721722 (+ 2497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:50.725615 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:50.725628 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:50.725632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:50.726203 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:50.726656 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:50.726661 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:50.726663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:50.726852 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:50.727490 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:50.727495 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:50.727675 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 02:12:50.727841 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 02:12:50.728161 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260926 02:12:50.728592 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.712667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39096 (local address 127.18.159.189:34883)
0926 02:12:50.714231 (+ 1564us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:50.714236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:50.714276 (+ 40us) server_negotiation.cc:407] Connection header received
0926 02:12:50.714342 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:50.714348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:50.714431 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:50.714538 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:50.715359 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.716030 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.717268 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.717688 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.717836 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.721997 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:50.722005 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:50.722011 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:50.722064 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:50.725446 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.726363 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.726368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.726371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.726520 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.727014 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.727018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.727021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.727343 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:50.727524 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:50.727941 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 02:12:50.728211 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1392}
I20260926 02:12:50.729501 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.712879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50446 (local address 127.18.159.190:36725)
0926 02:12:50.713127 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:50.713133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:50.713154 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:12:50.713356 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:50.713362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:50.713457 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:50.713573 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:50.715155 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.715875 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.717242 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.717657 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.717799 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.722935 (+ 5136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:50.722941 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:50.722944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:50.722981 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:50.726459 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.727293 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.727299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.727303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.727442 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.727829 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.727835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.727838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.728108 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:50.728446 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:50.728962 (+ 516us) server_negotiation.cc:299] Negotiation successful
0926 02:12:50.729215 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 02:12:50.729882 9251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:12:50.732228 9102 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } attempt: 4
Sep 26 02:12:50 dist-test-slave-03rn krb5kdc[9022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388770, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.188@KRBTEST.COM for kudu/127.18.159.188@KRBTEST.COM
Sep 26 02:12:50 dist-test-slave-03rn krb5kdc[9022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.189@KRBTEST.COM
Sep 26 02:12:50 dist-test-slave-03rn krb5kdc[9022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388769, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for kudu/127.18.159.188@KRBTEST.COM
I20260926 02:12:50.745424 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.730703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:39509 (local address 127.0.0.1:51944)
0926 02:12:50.730950 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.730963 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:50.731051 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:50.731366 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:50.731371 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:50.731778 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:50.731917 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:50.731925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.733092 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.733096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:50.734567 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:50.734586 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.735237 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.735242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:50.735484 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.736121 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:50.736139 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:50.738887 (+ 2748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:50.742498 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:50.742503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:50.742507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:50.742991 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:50.743432 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:50.743436 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:50.743441 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:50.743604 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:50.744310 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:50.744315 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:50.744527 (+ 212us) client_negotiation.cc:769] Sending connection context
0926 02:12:50.744729 (+ 202us) client_negotiation.cc:240] Negotiation successful
0926 02:12:50.745093 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260926 02:12:50.746080 9254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.730714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51944 (local address 127.18.159.188:39509)
0926 02:12:50.730879 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:50.730883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:50.731037 (+ 154us) server_negotiation.cc:407] Connection header received
0926 02:12:50.731184 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:50.731187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:50.731242 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:50.731352 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:50.732048 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.732942 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.734704 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.735089 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.735230 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.739093 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:50.739099 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:50.739105 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:50.739144 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:50.742340 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.743123 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.743130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.743134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.743281 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.743741 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.743746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.743749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.744156 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:50.744338 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:50.744863 (+ 525us) server_negotiation.cc:299] Negotiation successful
0926 02:12:50.745647 (+ 784us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":486,"server-negotiator.queue_time_us":54}
I20260926 02:12:50.750195 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.732836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:39509 (local address 127.0.0.1:51950)
0926 02:12:50.733042 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.733061 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:50.733196 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:50.734402 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:50.734409 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:50.734930 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:50.735129 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:50.735141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.736739 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.736745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:50.738067 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:50.738086 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.739229 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.739237 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:50.739481 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.740144 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:50.740202 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:50.743085 (+ 2883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:50.747053 (+ 3968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:50.747059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:50.747063 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:50.747612 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:50.748085 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:50.748090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:50.748095 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:50.748333 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:50.749041 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:50.749047 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:50.749239 (+ 192us) client_negotiation.cc:769] Sending connection context
0926 02:12:50.749403 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 02:12:50.749723 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 02:12:50.750196 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.733288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51950 (local address 127.18.159.188:39509)
0926 02:12:50.734094 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:50.734099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:50.734127 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:12:50.734168 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:50.734172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:50.734237 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:50.734339 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:50.735328 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.736610 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.738249 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.739002 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.739788 (+ 786us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.743282 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:50.743287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:50.743290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:50.743317 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:50.746898 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.747764 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.747770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.747776 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.747943 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.748480 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.748485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.748488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.748891 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:50.749060 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:50.749522 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 02:12:50.749763 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":490,"threads_started":1}
I20260926 02:12:50.751155 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.731432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.18.159.189:34883)
0926 02:12:50.731595 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:50.731601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:50.731709 (+ 108us) server_negotiation.cc:407] Connection header received
0926 02:12:50.731894 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:50.731897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:50.731953 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:50.732060 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:50.733736 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.734658 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.736476 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.737138 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.737270 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.742794 (+ 5524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:50.742808 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:50.742811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:50.742847 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:50.747217 (+ 4370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.748129 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.748134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.748136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.748339 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.748750 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.748755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.748758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.749090 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:50.749384 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:50.750554 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0926 02:12:50.750841 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 02:12:50.752312 9251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6: Bootstrap starting.
I20260926 02:12:50.756093 9251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:50.756361 9102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d: Bootstrap starting.
I20260926 02:12:50.756929 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:12:50.757543 9251 log.cc:824] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:50.760229 9102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:50.760999 9251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6: No bootstrap required, opened a new log
I20260926 02:12:50.761677 9102 log.cc:824] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:50.760345 9077 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "38b26fd932eb477996c55ba41b9c4d8d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50446
I20260926 02:12:50.765394 9102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d: No bootstrap required, opened a new log
I20260926 02:12:50.769039 9251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b26fd932eb477996c55ba41b9c4d8d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36725 } } peers { permanent_uuid: "118d2ea9f5c345a08ad712288f07b3fc" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } } peers { permanent_uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } }
I20260926 02:12:50.769454 9251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ffc78a4fbbd4ffba745ba4ab9e61cd6, State: Initialized, Role: FOLLOWER
I20260926 02:12:50.769922 9251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b26fd932eb477996c55ba41b9c4d8d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36725 } } peers { permanent_uuid: "118d2ea9f5c345a08ad712288f07b3fc" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } } peers { permanent_uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } }
I20260926 02:12:50.770864 9260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b26fd932eb477996c55ba41b9c4d8d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36725 } } peers { permanent_uuid: "118d2ea9f5c345a08ad712288f07b3fc" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } } peers { permanent_uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } } }
I20260926 02:12:50.771258 9260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:12:50.772091 9251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:50.773288 9102 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b26fd932eb477996c55ba41b9c4d8d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36725 } } peers { permanent_uuid: "118d2ea9f5c345a08ad712288f07b3fc" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } } peers { permanent_uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } }
I20260926 02:12:50.773779 9102 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b26fd932eb477996c55ba41b9c4d8d, State: Initialized, Role: FOLLOWER
I20260926 02:12:50.774290 9102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b26fd932eb477996c55ba41b9c4d8d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36725 } } peers { permanent_uuid: "118d2ea9f5c345a08ad712288f07b3fc" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } } peers { permanent_uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } }
I20260926 02:12:50.775586 9266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b26fd932eb477996c55ba41b9c4d8d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36725 } } peers { permanent_uuid: "118d2ea9f5c345a08ad712288f07b3fc" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } } peers { permanent_uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } } }
I20260926 02:12:50.776391 9266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:12:50.776844 9102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:50.776954 9077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 02:12:50.777138 9077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:50.781695 9077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b26fd932eb477996c55ba41b9c4d8d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36725 } } peers { permanent_uuid: "118d2ea9f5c345a08ad712288f07b3fc" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } } peers { permanent_uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } }
I20260926 02:12:50.785226 9226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38b26fd932eb477996c55ba41b9c4d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6"
I20260926 02:12:50.785979 9226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:50.787124 9077 leader_election.cc:288] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [CANDIDATE]: Term 1 election: Requested vote from peers 118d2ea9f5c345a08ad712288f07b3fc (127.18.159.189:34883), 5ffc78a4fbbd4ffba745ba4ab9e61cd6 (127.18.159.188:39509)
I20260926 02:12:50.805132 9226 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38b26fd932eb477996c55ba41b9c4d8d in term 1.
I20260926 02:12:50.807542 9051 leader_election.cc:302] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38b26fd932eb477996c55ba41b9c4d8d, 5ffc78a4fbbd4ffba745ba4ab9e61cd6; no voters:
I20260926 02:12:50.808288 9266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [term 1 FOLLOWER]: Leader election won for term 1
W20260926 02:12:50.818099 9272 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:12:50.818239 9272 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:12:50.780202 9184 master_runner.cc:186] Error getting master registration for 127.18.159.190:36725: OK, instance_id { permanent_uuid: "38b26fd932eb477996c55ba41b9c4d8d" instance_seqno: 1790388769590053 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:12:50.821842 9184 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:12:50.827004 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.794550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.18.159.190:36725)
0926 02:12:50.794765 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:50.794772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:50.795288 (+ 516us) server_negotiation.cc:407] Connection header received
0926 02:12:50.796332 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:50.796341 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:50.796448 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:50.796547 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:50.802188 (+ 5641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.803696 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.808763 (+ 5067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.809727 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.809873 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.817226 (+ 7353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:50.817236 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:50.817241 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:50.817293 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:50.821972 (+ 4679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.823724 (+ 1752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.823732 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.823736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.823937 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.824699 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.824717 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.824721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.825107 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:50.825402 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:50.826308 (+ 906us) server_negotiation.cc:299] Negotiation successful
0926 02:12:50.826582 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 02:12:50.828316 9056 tablet.cc:2369] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:12:50.830626 9266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [term 1 LEADER]: Becoming Leader. State: Replica: 38b26fd932eb477996c55ba41b9c4d8d, State: Running, Role: LEADER
I20260926 02:12:50.831435 9266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b26fd932eb477996c55ba41b9c4d8d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36725 } } peers { permanent_uuid: "118d2ea9f5c345a08ad712288f07b3fc" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } } peers { permanent_uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } }
I20260926 02:12:50.846212 9286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38b26fd932eb477996c55ba41b9c4d8d. Latest consensus state: current_term: 1 leader_uuid: "38b26fd932eb477996c55ba41b9c4d8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b26fd932eb477996c55ba41b9c4d8d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36725 } } peers { permanent_uuid: "118d2ea9f5c345a08ad712288f07b3fc" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } } peers { permanent_uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } } }
I20260926 02:12:50.846846 9286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:50.849356 9290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:50.858827 9176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } attempt: 5
I20260926 02:12:50.864683 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.794054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39108 (local address 127.18.159.189:34883)
0926 02:12:50.794268 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:50.794277 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:50.828968 (+ 34691us) server_negotiation.cc:407] Connection header received
0926 02:12:50.829397 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:50.829403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:50.829496 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:50.829581 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:50.832908 (+ 3327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.834032 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.837150 (+ 3118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.837886 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.838001 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.851090 (+ 13089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:50.851099 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:50.851104 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:50.851178 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:50.855193 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.859534 (+ 4341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.859541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.859544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.859751 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.861207 (+ 1456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.861217 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.861222 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.861625 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:50.861872 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:50.863900 (+ 2028us) server_negotiation.cc:299] Negotiation successful
0926 02:12:50.864266 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":69}
Sep 26 02:12:50 dist-test-slave-03rn krb5kdc[9022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388770, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.189@KRBTEST.COM for kudu/127.18.159.188@KRBTEST.COM
I20260926 02:12:50.880096 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.795750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51958 (local address 127.18.159.188:39509)
0926 02:12:50.795928 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:50.795934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:50.838703 (+ 42769us) server_negotiation.cc:407] Connection header received
0926 02:12:50.839025 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:50.839068 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:50.839211 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:50.839302 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:50.844913 (+ 5611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.846104 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.858207 (+ 12103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.858939 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.859089 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.869873 (+ 10784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:50.869881 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:50.869886 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:50.869926 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:50.873896 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.875471 (+ 1575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.875478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.875508 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.875851 (+ 343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.876720 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.876751 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.876754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.877484 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:50.877876 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:50.878897 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0926 02:12:50.879409 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:12:50.880834 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.859480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:39509 (local address 127.0.0.1:51970)
0926 02:12:50.859767 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:50.859798 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:50.859906 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:50.860415 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:50.860420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:50.860974 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:50.861187 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:50.861200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.862758 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.862763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:50.864315 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:50.864332 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.865710 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.865716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:50.865978 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.866689 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:50.866715 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:50.869588 (+ 2873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:50.877627 (+ 8039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:50.877635 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:50.877639 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:50.878245 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:50.878746 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:50.878751 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:50.878755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:50.878923 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:50.879730 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:50.879735 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:50.879901 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 02:12:50.880076 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 02:12:50.880418 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":32}
I20260926 02:12:50.881896 9254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:50.859561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51970 (local address 127.18.159.188:39509)
0926 02:12:50.859734 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:50.859741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:50.859845 (+ 104us) server_negotiation.cc:407] Connection header received
0926 02:12:50.860011 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:50.860017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:50.860095 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:50.860230 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:50.861382 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.862556 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.864496 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:50.865216 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:50.865347 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:50.869826 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:50.869833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:50.869843 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:50.869890 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:50.877361 (+ 7471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.878404 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.878410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.878414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.878596 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:50.879092 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:50.879097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:50.879101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:50.879486 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:50.879666 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:50.881227 (+ 1561us) server_negotiation.cc:299] Negotiation successful
0926 02:12:50.881474 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 02:12:50.890019 9176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118d2ea9f5c345a08ad712288f07b3fc: Bootstrap starting.
I20260926 02:12:50.897519 9176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 118d2ea9f5c345a08ad712288f07b3fc: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:50.899113 9176 log.cc:824] T 00000000000000000000000000000000 P 118d2ea9f5c345a08ad712288f07b3fc: Log is configured to *not* fsync() on all Append() calls
W20260926 02:12:50.904292 9205 tablet.cc:2369] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:12:50.904649 9290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:50.920692 9176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118d2ea9f5c345a08ad712288f07b3fc: No bootstrap required, opened a new log
I20260926 02:12:50.926709 9297 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 02:12:50.936267 9226 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6 [term 1 FOLLOWER]: Refusing update from remote peer 38b26fd932eb477996c55ba41b9c4d8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:12:50.937552 9266 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:12:50.944000 9176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 118d2ea9f5c345a08ad712288f07b3fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b26fd932eb477996c55ba41b9c4d8d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36725 } } peers { permanent_uuid: "118d2ea9f5c345a08ad712288f07b3fc" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } } peers { permanent_uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } }
I20260926 02:12:50.944489 9176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 118d2ea9f5c345a08ad712288f07b3fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 118d2ea9f5c345a08ad712288f07b3fc, State: Initialized, Role: FOLLOWER
I20260926 02:12:50.945097 9176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 118d2ea9f5c345a08ad712288f07b3fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b26fd932eb477996c55ba41b9c4d8d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36725 } } peers { permanent_uuid: "118d2ea9f5c345a08ad712288f07b3fc" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } } peers { permanent_uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } }
I20260926 02:12:50.958386 9298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118d2ea9f5c345a08ad712288f07b3fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b26fd932eb477996c55ba41b9c4d8d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36725 } } peers { permanent_uuid: "118d2ea9f5c345a08ad712288f07b3fc" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } } peers { permanent_uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } } }
I20260926 02:12:50.959025 9298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118d2ea9f5c345a08ad712288f07b3fc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:12:50.960340 9176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 118d2ea9f5c345a08ad712288f07b3fc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:50.960772 9150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 118d2ea9f5c345a08ad712288f07b3fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:50.964365 9151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38b26fd932eb477996c55ba41b9c4d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "118d2ea9f5c345a08ad712288f07b3fc"
I20260926 02:12:50.938859 9035 master_runner.cc:186] Error getting master registration for 127.18.159.189:34883: OK, instance_id { permanent_uuid: "118d2ea9f5c345a08ad712288f07b3fc" instance_seqno: 1790388770120250 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:12:50.965551 9035 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:12:50.965754 9150 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 118d2ea9f5c345a08ad712288f07b3fc [term 1 FOLLOWER]: Refusing update from remote peer 38b26fd932eb477996c55ba41b9c4d8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:12:50.969689 9151 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 118d2ea9f5c345a08ad712288f07b3fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38b26fd932eb477996c55ba41b9c4d8d in term 1.
I20260926 02:12:50.972570 9266 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "118d2ea9f5c345a08ad712288f07b3fc" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:12:50.990561 9260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38b26fd932eb477996c55ba41b9c4d8d. Latest consensus state: current_term: 1 leader_uuid: "38b26fd932eb477996c55ba41b9c4d8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b26fd932eb477996c55ba41b9c4d8d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36725 } } peers { permanent_uuid: "118d2ea9f5c345a08ad712288f07b3fc" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } } peers { permanent_uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } } }
I20260926 02:12:50.990890 9260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:12:50.992564 9296 mvcc.cc:205] Tried to move back new op lower bound from 7333432405734862848 to 7333432405333569536. Current Snapshot: MvccSnapshot[applied={T|T < 7333432405734862848}]
I20260926 02:12:51.017768 9286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38b26fd932eb477996c55ba41b9c4d8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b26fd932eb477996c55ba41b9c4d8d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36725 } } peers { permanent_uuid: "118d2ea9f5c345a08ad712288f07b3fc" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } } peers { permanent_uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } } }
I20260926 02:12:51.018093 9286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:51.021766 9298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118d2ea9f5c345a08ad712288f07b3fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38b26fd932eb477996c55ba41b9c4d8d. Latest consensus state: current_term: 1 leader_uuid: "38b26fd932eb477996c55ba41b9c4d8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b26fd932eb477996c55ba41b9c4d8d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36725 } } peers { permanent_uuid: "118d2ea9f5c345a08ad712288f07b3fc" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } } peers { permanent_uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } } }
I20260926 02:12:51.022104 9298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118d2ea9f5c345a08ad712288f07b3fc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:12:51.034176 9315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38b26fd932eb477996c55ba41b9c4d8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b26fd932eb477996c55ba41b9c4d8d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36725 } } peers { permanent_uuid: "118d2ea9f5c345a08ad712288f07b3fc" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } } peers { permanent_uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } } }
I20260926 02:12:51.034780 9316 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 02:12:51.034906 9315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:51.035962 9298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118d2ea9f5c345a08ad712288f07b3fc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38b26fd932eb477996c55ba41b9c4d8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b26fd932eb477996c55ba41b9c4d8d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36725 } } peers { permanent_uuid: "118d2ea9f5c345a08ad712288f07b3fc" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } } peers { permanent_uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } } }
I20260926 02:12:51.036249 9298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118d2ea9f5c345a08ad712288f07b3fc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:12:51.038028 9260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38b26fd932eb477996c55ba41b9c4d8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b26fd932eb477996c55ba41b9c4d8d" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 36725 } } peers { permanent_uuid: "118d2ea9f5c345a08ad712288f07b3fc" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 34883 } } peers { permanent_uuid: "5ffc78a4fbbd4ffba745ba4ab9e61cd6" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 39509 } } }
I20260926 02:12:51.038286 9260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:12:51.040297 9290 catalog_manager.cc:1347] Generated new cluster ID: e73b1e8518af4425adacc77ff48af5b4
I20260926 02:12:51.040405 9290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:51.071713 9290 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:51.073832 9290 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:51.089819 9290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38b26fd932eb477996c55ba41b9c4d8d: Generated new TSK 0
I20260926 02:12:51.091051 9290 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 02:12:51.196218 9255 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:51.180912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50462 (local address 127.18.159.190:36725)
0926 02:12:51.181088 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:51.181094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:51.193251 (+ 12157us) server_negotiation.cc:407] Connection header received
0926 02:12:51.194527 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:51.194533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:51.194615 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:51.194712 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:51.195883 (+ 1171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50462: BlockingRecv error: recv got EOF from 127.0.0.1:50462 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 02:12:51.197965 9181 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:51.180963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39122 (local address 127.18.159.189:34883)
0926 02:12:51.181184 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:51.181192 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:51.193778 (+ 12586us) server_negotiation.cc:407] Connection header received
0926 02:12:51.194075 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:51.194082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:51.194183 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:51.194258 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:51.197729 (+ 3471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39122: BlockingRecv error: recv got EOF from 127.0.0.1:39122 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 02:12:51.200963 9254 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:51.194232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51986 (local address 127.18.159.188:39509)
0926 02:12:51.194415 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:51.194423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:51.198798 (+ 4375us) server_negotiation.cc:407] Connection header received
0926 02:12:51.199489 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:51.199497 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:51.199614 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:51.199702 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:51.200698 (+ 996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51986: BlockingRecv error: recv got EOF from 127.0.0.1:51986 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 02:12:51.210348 9181 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:51.207939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.18.159.189:34883)
0926 02:12:51.208202 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:51.208208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:51.208740 (+ 532us) server_negotiation.cc:407] Connection header received
0926 02:12:51.208974 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:51.208979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:51.209113 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:51.209238 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:51.210085 (+ 847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39136: BlockingRecv error: recv got EOF from 127.0.0.1:39136 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 02:12:51.217286 9181 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:51.215024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.18.159.189:34883)
0926 02:12:51.215217 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:51.215223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:51.215726 (+ 503us) server_negotiation.cc:407] Connection header received
0926 02:12:51.215924 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:51.215933 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:51.216037 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:51.216171 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:51.217073 (+ 902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39152: BlockingRecv error: recv got EOF from 127.0.0.1:39152 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 02:12:51.230635 9181 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:51.226317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39160 (local address 127.18.159.189:34883)
0926 02:12:51.226529 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:51.226536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:51.228858 (+ 2322us) server_negotiation.cc:407] Connection header received
0926 02:12:51.229129 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:51.229138 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:51.229243 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:51.229356 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:51.230377 (+ 1021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39160: BlockingRecv error: recv got EOF from 127.0.0.1:39160 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 02:12:51.249027 9181 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:51.246719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39162 (local address 127.18.159.189:34883)
0926 02:12:51.246947 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:51.246954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:51.247394 (+ 440us) server_negotiation.cc:407] Connection header received
0926 02:12:51.247608 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:51.247614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:51.247715 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:51.247836 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:51.248763 (+ 927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39162: BlockingRecv error: recv got EOF from 127.0.0.1:39162 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260926 02:12:51.265139 9181 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:51.262537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39168 (local address 127.18.159.189:34883)
0926 02:12:51.262744 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:51.262752 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:51.263339 (+ 587us) server_negotiation.cc:407] Connection header received
0926 02:12:51.263592 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:51.263599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:51.263703 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:51.263829 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:51.264890 (+ 1061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39168: BlockingRecv error: recv got EOF from 127.0.0.1:39168 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 02:12:51.288548 9181 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:51.285745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39178 (local address 127.18.159.189:34883)
0926 02:12:51.285983 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:51.285989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:51.286594 (+ 605us) server_negotiation.cc:407] Connection header received
0926 02:12:51.286820 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:51.286826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:51.286939 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:51.287076 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:51.288308 (+ 1232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39178: BlockingRecv error: recv got EOF from 127.0.0.1:39178 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260926 02:12:51.332873 9181 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:51.329961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39184 (local address 127.18.159.189:34883)
0926 02:12:51.330529 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:51.330535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:51.331115 (+ 580us) server_negotiation.cc:407] Connection header received
0926 02:12:51.331351 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:51.331357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:51.331468 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:51.331593 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:51.332582 (+ 989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39184: BlockingRecv error: recv got EOF from 127.0.0.1:39184 (error 108)
Metrics: {"server-negotiator.queue_time_us":416}
W20260926 02:12:51.401986 9181 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:51.400320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39188 (local address 127.18.159.189:34883)
0926 02:12:51.400503 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:51.400507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:51.400864 (+ 357us) server_negotiation.cc:407] Connection header received
0926 02:12:51.401028 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:51.401031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:51.401115 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:51.401237 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:51.401827 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39188: BlockingRecv error: recv got EOF from 127.0.0.1:39188 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 02:12:51.539012 9181 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:51.536947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39204 (local address 127.18.159.189:34883)
0926 02:12:51.537191 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:51.537198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:51.537499 (+ 301us) server_negotiation.cc:407] Connection header received
0926 02:12:51.537749 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:51.537753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:51.537825 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:51.537937 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:51.538717 (+ 780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39204: BlockingRecv error: recv got EOF from 127.0.0.1:39204 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260926 02:12:51.800814 9181 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:51.798776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39210 (local address 127.18.159.189:34883)
0926 02:12:51.799005 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:51.799013 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:51.799402 (+ 389us) server_negotiation.cc:407] Connection header received
0926 02:12:51.799651 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:51.799656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:51.799732 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:51.799845 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:51.800648 (+ 803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39210: BlockingRecv error: recv got EOF from 127.0.0.1:39210 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 02:12:51.820377 9272 catalog_manager.cc:1259] Loaded cluster ID: e73b1e8518af4425adacc77ff48af5b4
I20260926 02:12:51.820487 9272 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6: loading cluster ID for follower catalog manager: success
I20260926 02:12:51.823082 9272 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6: acquiring CA information for follower catalog manager: success
I20260926 02:12:51.824927 9272 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5ffc78a4fbbd4ffba745ba4ab9e61cd6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:12:52.039769 9316 catalog_manager.cc:1259] Loaded cluster ID: e73b1e8518af4425adacc77ff48af5b4
I20260926 02:12:52.039921 9316 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 118d2ea9f5c345a08ad712288f07b3fc: loading cluster ID for follower catalog manager: success
I20260926 02:12:52.042951 9316 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 118d2ea9f5c345a08ad712288f07b3fc: acquiring CA information for follower catalog manager: success
I20260926 02:12:52.045008 9316 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 118d2ea9f5c345a08ad712288f07b3fc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:12:52.328871 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:52.321019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39218 (local address 127.18.159.189:34883)
0926 02:12:52.321892 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:52.321897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:52.321916 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:12:52.321982 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:52.321986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:52.322158 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:52.322279 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:52.322860 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:52.323833 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:52.326028 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:52.326552 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:52.326682 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:52.327321 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:52.327947 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:52.328074 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:52.328345 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 02:12:52.328444 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":602,"threads_started":1}
I20260926 02:12:52.346865 9389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:52.340791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51998 (local address 127.18.159.188:39509)
0926 02:12:52.341505 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:52.341511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:52.341535 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:12:52.341675 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:52.341679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:52.341830 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:52.342018 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:52.342447 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:52.343197 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:52.344571 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:52.344983 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:52.345155 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:52.345640 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:52.346208 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:52.346380 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:52.346468 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 02:12:52.346563 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":474,"threads_started":1}
I20260926 02:12:52.378010 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:52.354132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39220 (local address 127.18.159.189:34883)
0926 02:12:52.354320 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:52.354326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:52.366788 (+ 12462us) server_negotiation.cc:407] Connection header received
0926 02:12:52.367677 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:52.367682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:52.367769 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:52.367829 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:52.369095 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:52.370821 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:52.375144 (+ 4323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:52.375845 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:52.375954 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:52.377054 (+ 1100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:52.377228 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:52.377315 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:52.377577 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 02:12:52.377671 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:12:52.383910 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:52.353922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.18.159.190:36725)
0926 02:12:52.364660 (+ 10738us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:52.364670 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:52.365766 (+ 1096us) server_negotiation.cc:407] Connection header received
0926 02:12:52.365935 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:52.365944 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:52.366227 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:52.366432 (+ 205us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:52.367547 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:52.369515 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:52.380987 (+ 11472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:52.381699 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:52.381841 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:52.382429 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:52.382874 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:52.382965 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:52.383503 (+ 538us) server_negotiation.cc:299] Negotiation successful
0926 02:12:52.383592 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10534,"thread_start_us":10395,"threads_started":1}
I20260926 02:12:52.385772 9389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:52.370418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.18.159.188:39509)
0926 02:12:52.378743 (+ 8325us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:52.378751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:52.378775 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:12:52.378855 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:52.378863 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:52.378963 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:52.379088 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:52.380094 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:52.381178 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:52.383216 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:52.383930 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:52.384074 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:52.384747 (+ 673us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:52.384879 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:52.384966 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:52.385389 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 02:12:52.385473 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 02:12:52.395287 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 9035
I20260926 02:12:52.396503 9097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:52.509236 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 9035
I20260926 02:12:52.534138 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 9109
I20260926 02:12:52.535207 9171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:52.641561 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 9109
I20260926 02:12:52.661386 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 9184
I20260926 02:12:52.662422 9246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:52.773270 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 9184
2026-09-26T02:12:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5045 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:12:52 dist-test-slave-03rn krb5kdc[9403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:12:52 dist-test-slave-03rn krb5kdc[9403](info): set up 2 sockets
Sep 26 02:12:52 dist-test-slave-03rn krb5kdc[9403](info): commencing operation
krb5kdc: starting...
W20260926 02:12:53.901687 19070 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/test-user.keytab.
Sep 26 02:12:54 dist-test-slave-03rn krb5kdc[9403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:12:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:12:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.159.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
Entry for principal kudu/127.18.159.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab.
I20260926 02:12:54.180037 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:35193
--webserver_interface=127.18.159.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:35193,127.18.159.189:41019,127.18.159.188:43089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:12:54.534436 9416 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:12:54.534847 9416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:54.534986 9416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:54.535117 9416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:54.542716 9416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:54.542824 9416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:54.542935 9416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:54.543018 9416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:54.543111 9416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:54.553431 9416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:35193,127.18.159.189:41019,127.18.159.188:43089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.190.keytab
--principal=kudu/127.18.159.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.190:35193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.159.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:54.554813 9416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:54.556255 9416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:54.569156 9424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:54.569278 9421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:54.569250 9422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:54.570479 9416 server_base.cc:1034] running on GCE node
I20260926 02:12:54.571393 9416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:54.572979 9416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:54.574240 9416 hybrid_clock.cc:648] HybridClock initialized: now 1790388774574150 us; error 70 us; skew 500 ppm
I20260926 02:12:54.574602 9416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:54 dist-test-slave-03rn krb5kdc[9403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388774, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:54.577950 9416 init.cc:377] Logged in from keytab as kudu/127.18.159.190@KRBTEST.COM (short username kudu)
I20260926 02:12:54.579530 9416 webserver.cc:460] Webserver started at http://127.18.159.190:41061/ using document root <none> and password file <none>
I20260926 02:12:54.580317 9416 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:54.580447 9416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:54.580787 9416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:54.583810 9416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-0/data/instance:
uuid: "c2f9dd387af14051b54f0c52700df2b7"
format_stamp: "Formatted at 2026-09-26 02:12:54 on dist-test-slave-03rn"
server_key: "56faac89dbd8b91ab38c4b6d27c6dd88"
server_key_iv: "69c57c6c9d969c5de3be51fa7cae9e56"
server_key_version: "encryptionkey@0"
I20260926 02:12:54.584666 9416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-0/wal/instance:
uuid: "c2f9dd387af14051b54f0c52700df2b7"
format_stamp: "Formatted at 2026-09-26 02:12:54 on dist-test-slave-03rn"
server_key: "56faac89dbd8b91ab38c4b6d27c6dd88"
server_key_iv: "69c57c6c9d969c5de3be51fa7cae9e56"
server_key_version: "encryptionkey@0"
I20260926 02:12:54.590062 9416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:12:54.593734 9431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:54.595175 9416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:12:54.595392 9416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-0/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-0/wal
uuid: "c2f9dd387af14051b54f0c52700df2b7"
format_stamp: "Formatted at 2026-09-26 02:12:54 on dist-test-slave-03rn"
server_key: "56faac89dbd8b91ab38c4b6d27c6dd88"
server_key_iv: "69c57c6c9d969c5de3be51fa7cae9e56"
server_key_version: "encryptionkey@0"
I20260926 02:12:54.595652 9416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:54.613016 9416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:54.617527 9416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:54.617836 9416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:54.637964 9416 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.190:35193
I20260926 02:12:54.637970 9482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.190:35193 every 8 connection(s)
I20260926 02:12:54.639467 9416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-0/data/info.pb
I20260926 02:12:54.643616 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:54.647918 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 9416
I20260926 02:12:54.648162 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-0/wal/instance
I20260926 02:12:54.648682 19070 external_mini_cluster.cc:1428] Setting key 7cd086a3f1f2933099a661470decf7a2
I20260926 02:12:54.650633 9483 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35193 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:12:54 dist-test-slave-03rn krb5kdc[9403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388774, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.159.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.189@KRBTEST.COM" created.
I20260926 02:12:54.675170 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:54.654005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35193 (local address 127.0.0.1:52426)
0926 02:12:54.655059 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:54.655131 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:54.657161 (+ 2030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:54.658096 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:54.658103 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:54.658865 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:54.659227 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:54.659246 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:54.660894 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:54.660903 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:54.662005 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:54.662017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:54.662972 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:54.662978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:54.663147 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:54.663914 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:54.663939 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:54.666787 (+ 2848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:54.670431 (+ 3644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:54.670446 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:54.670460 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:54.670910 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:54.671369 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:54.671377 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:54.671383 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:54.671556 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:54.672474 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:54.672481 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:54.672726 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 02:12:54.672976 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 02:12:54.673720 (+ 744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":180,"thread_start_us":431,"threads_started":1}
I20260926 02:12:54.675192 9486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:54.654005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52426 (local address 127.18.159.190:35193)
0926 02:12:54.655753 (+ 1748us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:54.655763 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:54.655820 (+ 57us) server_negotiation.cc:407] Connection header received
0926 02:12:54.657579 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:54.657586 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:54.657861 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:54.658084 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:54.659452 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:54.660745 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:54.662155 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:54.662722 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:54.662928 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:54.667052 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:54.667062 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:54.667076 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:54.667107 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:54.670252 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:54.671060 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:54.671076 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:54.671079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:54.671249 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:54.671666 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:54.671674 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:54.671678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:54.672324 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:54.672542 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:54.673130 (+ 588us) server_negotiation.cc:299] Negotiation successful
0926 02:12:54.673589 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":1529,"spinlock_wait_cycles":36224,"thread_start_us":1308,"threads_started":1}
I20260926 02:12:54.678637 9483 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:12:54.679837 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:54.679343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:41019 (local address 127.0.0.1:47476)
0926 02:12:54.679555 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:54.679704 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260926 02:12:54.680150 9432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.159.189:41019: connect: Connection refused (error 111)
I20260926 02:12:54.681918 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:54.681510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:41019 (local address 127.0.0.1:47480)
0926 02:12:54.681688 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:54.681775 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 02:12:54.682116 9483 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.189:41019: Network error: Client connection negotiation failed: client connection to 127.18.159.189:41019: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.189.keytab.
Entry for principal kudu/127.18.159.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.189.keytab.
I20260926 02:12:54.707907 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.189:41019
--webserver_interface=127.18.159.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.189.keytab
--principal=kudu/127.18.159.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:35193,127.18.159.189:41019,127.18.159.188:43089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:12:54.731448 9483 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } attempt: 1
I20260926 02:12:54.732620 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:54.732137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:41019 (local address 127.0.0.1:47486)
0926 02:12:54.732362 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:54.732480 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 02:12:54.733709 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:54.733349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:41019 (local address 127.0.0.1:47488)
0926 02:12:54.733500 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:54.733576 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 02:12:54.733928 9483 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.189:41019: Network error: Client connection negotiation failed: client connection to 127.18.159.189:41019: connect: Connection refused (error 111)
I20260926 02:12:54.802208 9483 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } attempt: 2
I20260926 02:12:54.803601 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:54.803088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:41019 (local address 127.0.0.1:47500)
0926 02:12:54.803353 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:54.803443 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 02:12:54.804816 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:54.804424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:41019 (local address 127.0.0.1:47502)
0926 02:12:54.804601 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:54.804672 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 02:12:54.805002 9483 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.189:41019: Network error: Client connection negotiation failed: client connection to 127.18.159.189:41019: connect: Connection refused (error 111)
I20260926 02:12:54.896318 9483 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } attempt: 3
I20260926 02:12:54.897539 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:54.897101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:41019 (local address 127.0.0.1:47506)
0926 02:12:54.897305 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:54.897399 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 02:12:54.898653 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:54.898285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:41019 (local address 127.0.0.1:47512)
0926 02:12:54.898450 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:54.898522 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 02:12:54.898838 9483 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.189:41019: Network error: Client connection negotiation failed: client connection to 127.18.159.189:41019: connect: Connection refused (error 111)
I20260926 02:12:55.042126 9483 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } attempt: 4
I20260926 02:12:55.043431 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.042889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:41019 (local address 127.0.0.1:47524)
0926 02:12:55.043201 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.043290 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260926 02:12:55.044564 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.044140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:41019 (local address 127.0.0.1:47534)
0926 02:12:55.044350 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.044438 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.189:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260926 02:12:55.044751 9483 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.189:41019: Network error: Client connection negotiation failed: client connection to 127.18.159.189:41019: connect: Connection refused (error 111)
W20260926 02:12:55.063670 9490 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:12:55.064111 9490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:55.064265 9490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:55.064409 9490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:55.071970 9490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:55.072146 9490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:55.072296 9490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:55.072386 9490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:55.072482 9490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:55.082988 9490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:35193,127.18.159.189:41019,127.18.159.188:43089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.189.keytab
--principal=kudu/127.18.159.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.189:41019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.159.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:55.084571 9490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:55.085991 9490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:55.098526 9495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:55.098613 9496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:55.098737 9498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:55.099881 9490 server_base.cc:1034] running on GCE node
I20260926 02:12:55.100906 9490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:55.102557 9490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:55.103837 9490 hybrid_clock.cc:648] HybridClock initialized: now 1790388775103773 us; error 52 us; skew 500 ppm
I20260926 02:12:55.104292 9490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:55 dist-test-slave-03rn krb5kdc[9403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388775, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:55.107735 9490 init.cc:377] Logged in from keytab as kudu/127.18.159.189@KRBTEST.COM (short username kudu)
I20260926 02:12:55.109444 9490 webserver.cc:460] Webserver started at http://127.18.159.189:38935/ using document root <none> and password file <none>
I20260926 02:12:55.110229 9490 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:55.110373 9490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:55.110723 9490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:55.114209 9490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-1/data/instance:
uuid: "606b83010d4346569e3eab64700312d9"
format_stamp: "Formatted at 2026-09-26 02:12:55 on dist-test-slave-03rn"
server_key: "42e2314c55004753c6e18086cb8e1a8d"
server_key_iv: "1ff98208f1c42842230c8c091d79593c"
server_key_version: "encryptionkey@0"
I20260926 02:12:55.115157 9490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-1/wal/instance:
uuid: "606b83010d4346569e3eab64700312d9"
format_stamp: "Formatted at 2026-09-26 02:12:55 on dist-test-slave-03rn"
server_key: "42e2314c55004753c6e18086cb8e1a8d"
server_key_iv: "1ff98208f1c42842230c8c091d79593c"
server_key_version: "encryptionkey@0"
I20260926 02:12:55.121160 9490 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 02:12:55.125017 9505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:55.126530 9490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:12:55.126768 9490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-1/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-1/wal
uuid: "606b83010d4346569e3eab64700312d9"
format_stamp: "Formatted at 2026-09-26 02:12:55 on dist-test-slave-03rn"
server_key: "42e2314c55004753c6e18086cb8e1a8d"
server_key_iv: "1ff98208f1c42842230c8c091d79593c"
server_key_version: "encryptionkey@0"
I20260926 02:12:55.127007 9490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:55.149844 9490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:55.154737 9490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:55.155040 9490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:55.174657 9490 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.189:41019
I20260926 02:12:55.174679 9556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.189:41019 every 8 connection(s)
I20260926 02:12:55.176316 9490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-1/data/info.pb
I20260926 02:12:55.180228 9557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:55.186693 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 9490
I20260926 02:12:55.186905 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-1/wal/instance
I20260926 02:12:55.187340 19070 external_mini_cluster.cc:1428] Setting key 68c81b667f2a6d79eccbaaace1a430a7
I20260926 02:12:55.187146 9557 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35193 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:12:55 dist-test-slave-03rn krb5kdc[9403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388775, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.189@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:12:55.210721 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.190034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52442 (local address 127.18.159.190:35193)
0926 02:12:55.190535 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:55.190539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:55.191619 (+ 1080us) server_negotiation.cc:407] Connection header received
0926 02:12:55.193683 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:55.193690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:55.193796 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:55.193925 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:55.195590 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.196380 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.199168 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.199686 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.199821 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.204290 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:55.204296 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:55.204300 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:55.204349 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:55.207186 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.208057 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.208065 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.208074 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.208274 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.208823 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.208829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.208832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.209139 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:55.209308 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:55.210090 (+ 782us) server_negotiation.cc:299] Negotiation successful
0926 02:12:55.210339 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":289,"threads_started":1}
I20260926 02:12:55.211838 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.190654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35193 (local address 127.0.0.1:52442)
0926 02:12:55.191447 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.191486 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:55.193386 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:55.194186 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:55.194195 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:55.194761 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:55.195341 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:55.195359 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.196551 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.196555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:55.198788 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:55.198807 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.199950 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.199959 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:55.200355 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.201154 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:55.201189 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:55.203974 (+ 2785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:55.207380 (+ 3406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:55.207391 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:55.207411 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:55.207916 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:55.208395 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:55.208400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:55.208404 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:55.208690 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:55.209294 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:55.209303 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:55.209706 (+ 403us) client_negotiation.cc:769] Sending connection context
0926 02:12:55.209995 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 02:12:55.210563 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":318,"threads_started":1}
WARNING: no policy specified for kudu/127.18.159.188@KRBTEST.COM; defaulting to no policy
I20260926 02:12:55.213691 9557 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.159.188@KRBTEST.COM" created.
Sep 26 02:12:55 dist-test-slave-03rn krb5kdc[9403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388775, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.189@KRBTEST.COM for kudu/127.18.159.189@KRBTEST.COM
I20260926 02:12:55.230943 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.214373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:41019 (local address 127.0.0.1:47546)
0926 02:12:55.214553 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.214578 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:55.214679 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:55.216085 (+ 1406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:55.216089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:55.216546 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:55.216686 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:55.216694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.218754 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.218757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:55.219625 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:55.219636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.220919 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.220927 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:55.221163 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.221806 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:55.221824 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:55.224236 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:55.227750 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:55.227758 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:55.227765 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:55.228374 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:55.228814 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:55.228821 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:55.228827 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:55.229013 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:55.229659 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:55.229667 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:55.229835 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 02:12:55.229980 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 02:12:55.230326 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":3}
I20260926 02:12:55.230968 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.214734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47546 (local address 127.18.159.189:41019)
0926 02:12:55.215483 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:55.215493 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:55.215530 (+ 37us) server_negotiation.cc:407] Connection header received
0926 02:12:55.215623 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:55.215632 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:55.215911 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:55.216075 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:55.216871 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.218615 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.219769 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.220760 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.220899 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.224481 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:55.224504 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:55.224525 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:55.224568 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:55.227553 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.228537 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.228544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.228546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.228677 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.229131 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.229134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.229136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.229521 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:55.229691 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:55.230131 (+ 440us) server_negotiation.cc:299] Negotiation successful
0926 02:12:55.230519 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":197,"server-negotiator.queue_time_us":567,"thread_start_us":421,"threads_started":1}
I20260926 02:12:55.233665 9557 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:12:55.234997 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.234418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:43089 (local address 127.0.0.1:50766)
0926 02:12:55.234644 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.234785 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 02:12:55.235337 9507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
I20260926 02:12:55.237134 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.236651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:43089 (local address 127.0.0.1:50772)
0926 02:12:55.236861 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.236970 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 02:12:55.237416 9557 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:43089: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.159.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.188.keytab.
Entry for principal kudu/127.18.159.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.188.keytab.
I20260926 02:12:55.252781 19070 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8FmYUC/build/asan/bin/kudu
/tmp/dist-test-task8FmYUC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.188:43089
--webserver_interface=127.18.159.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.188.keytab
--principal=kudu/127.18.159.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.159.148:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.159.190:35193,127.18.159.189:41019,127.18.159.188:43089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:12:55.286798 9557 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } attempt: 1
I20260926 02:12:55.288084 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.287530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:43089 (local address 127.0.0.1:50782)
0926 02:12:55.287761 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.287892 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 02:12:55.289330 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.288873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:43089 (local address 127.0.0.1:50788)
0926 02:12:55.289073 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.289173 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 02:12:55.289585 9557 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:43089: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
I20260926 02:12:55.344026 9483 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } attempt: 5
Sep 26 02:12:55 dist-test-slave-03rn krb5kdc[9403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388774, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for kudu/127.18.159.189@KRBTEST.COM
I20260926 02:12:55.357810 9557 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } attempt: 2
I20260926 02:12:55.358855 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.358389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:43089 (local address 127.0.0.1:50794)
0926 02:12:55.358578 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.358671 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 02:12:55.359485 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.344826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:41019 (local address 127.0.0.1:47560)
0926 02:12:55.345063 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.345091 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:55.345265 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:55.345729 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:55.345734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:55.346293 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:55.346528 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:55.346541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.347550 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.347558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:55.349132 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:55.349149 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.349908 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.349914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:55.350127 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.350799 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:55.350823 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:55.353300 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:55.356710 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:55.356716 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:55.356720 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:55.357155 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:55.357575 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:55.357581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:55.357585 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:55.357759 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:55.358513 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:55.358519 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:55.358689 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 02:12:55.358800 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 02:12:55.359149 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 02:12:55.359745 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.344956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47560 (local address 127.18.159.189:41019)
0926 02:12:55.345155 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:55.345160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:55.345181 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:12:55.345458 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:55.345463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:55.345553 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:55.345645 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:55.346696 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.347394 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.349297 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.349747 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.349848 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.353529 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:55.353534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:55.353537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:55.353574 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:55.356503 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.357297 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.357304 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.357307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.357445 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.357914 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.357920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.357923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.358351 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:55.359098 (+ 747us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:55.359255 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 02:12:55.359485 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 02:12:55.360702 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.360323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:43089 (local address 127.0.0.1:50802)
0926 02:12:55.360474 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.360560 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 02:12:55.360749 9483 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } has no permanent_uuid. Determining permanent_uuid...
W20260926 02:12:55.361305 9557 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:43089: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
I20260926 02:12:55.361879 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.361431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:43089 (local address 127.0.0.1:50812)
0926 02:12:55.361590 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.361692 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 02:12:55.363039 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.362573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:43089 (local address 127.0.0.1:50828)
0926 02:12:55.362759 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.362877 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 02:12:55.363304 9483 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:43089: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
I20260926 02:12:55.414592 9483 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } attempt: 1
I20260926 02:12:55.415884 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.415392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:43089 (local address 127.0.0.1:50830)
0926 02:12:55.415614 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.415713 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 02:12:55.417068 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.416746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:43089 (local address 127.0.0.1:50832)
0926 02:12:55.416877 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.416951 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 02:12:55.417366 9483 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:43089: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
I20260926 02:12:55.452595 9557 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } attempt: 3
I20260926 02:12:55.453895 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.453338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:43089 (local address 127.0.0.1:50842)
0926 02:12:55.453568 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.453680 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 02:12:55.455114 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.454614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:43089 (local address 127.0.0.1:50846)
0926 02:12:55.454878 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.454975 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
W20260926 02:12:55.455343 9557 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:43089: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
I20260926 02:12:55.485637 9483 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } attempt: 2
I20260926 02:12:55.486948 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.486428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:43089 (local address 127.0.0.1:50858)
0926 02:12:55.486670 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.486766 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 02:12:55.487999 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.487679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:43089 (local address 127.0.0.1:50864)
0926 02:12:55.487811 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.487892 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 02:12:55.488292 9483 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:43089: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
I20260926 02:12:55.594511 9483 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } attempt: 3
I20260926 02:12:55.595783 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.595281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:43089 (local address 127.0.0.1:50868)
0926 02:12:55.595500 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.595594 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 02:12:55.597072 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.596604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:43089 (local address 127.0.0.1:50882)
0926 02:12:55.596834 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.596935 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260926 02:12:55.597329 9483 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:43089: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
I20260926 02:12:55.598580 9557 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } attempt: 4
I20260926 02:12:55.599951 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.599289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:43089 (local address 127.0.0.1:50884)
0926 02:12:55.599601 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.599742 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260926 02:12:55.601204 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.600760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:43089 (local address 127.0.0.1:50896)
0926 02:12:55.600941 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.601043 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 02:12:55.601461 9557 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.159.188:43089: Network error: Client connection negotiation failed: client connection to 127.18.159.188:43089: connect: Connection refused (error 111)
W20260926 02:12:55.615439 9565 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:12:55.615849 9565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:12:55.615957 9565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:12:55.616101 9565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:12:55.624913 9565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:12:55.625021 9565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:12:55.625131 9565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:12:55.625232 9565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:12:55.625358 9565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:12:55.636929 9565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.159.148:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.159.190:35193,127.18.159.189:41019,127.18.159.188:43089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/krb5kdc/kudu_127.18.159.188.keytab
--principal=kudu/127.18.159.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.159.188:43089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.159.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:12:55.638391 9565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:12:55.639914 9565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:12:55.653337 9570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:55.653607 9573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:12:55.653606 9571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:12:55.658080 9565 server_base.cc:1034] running on GCE node
I20260926 02:12:55.659077 9565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:12:55.660770 9565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:12:55.662048 9565 hybrid_clock.cc:648] HybridClock initialized: now 1790388775661987 us; error 48 us; skew 500 ppm
I20260926 02:12:55.662438 9565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:12:55 dist-test-slave-03rn krb5kdc[9403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388775, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:12:55.665899 9565 init.cc:377] Logged in from keytab as kudu/127.18.159.188@KRBTEST.COM (short username kudu)
I20260926 02:12:55.667469 9565 webserver.cc:460] Webserver started at http://127.18.159.188:41329/ using document root <none> and password file <none>
I20260926 02:12:55.668269 9565 fs_manager.cc:359] Metadata directory not provided
I20260926 02:12:55.668372 9565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:12:55.668663 9565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:12:55.672394 9565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-2/data/instance:
uuid: "8722e59c39f648579c296b291b50718b"
format_stamp: "Formatted at 2026-09-26 02:12:55 on dist-test-slave-03rn"
server_key: "538c98ff9a3e7580a41ae36c8a5e30ef"
server_key_iv: "15462181ea7f2d17315b1aba70623c6e"
server_key_version: "encryptionkey@0"
I20260926 02:12:55.673302 9565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-2/wal/instance:
uuid: "8722e59c39f648579c296b291b50718b"
format_stamp: "Formatted at 2026-09-26 02:12:55 on dist-test-slave-03rn"
server_key: "538c98ff9a3e7580a41ae36c8a5e30ef"
server_key_iv: "15462181ea7f2d17315b1aba70623c6e"
server_key_version: "encryptionkey@0"
I20260926 02:12:55.678838 9565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:12:55.682770 9580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:12:55.684419 9565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 02:12:55.684639 9565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-2/data,/tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-2/wal
uuid: "8722e59c39f648579c296b291b50718b"
format_stamp: "Formatted at 2026-09-26 02:12:55 on dist-test-slave-03rn"
server_key: "538c98ff9a3e7580a41ae36c8a5e30ef"
server_key_iv: "15462181ea7f2d17315b1aba70623c6e"
server_key_version: "encryptionkey@0"
I20260926 02:12:55.684897 9565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:12:55.703752 9565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:12:55.708776 9565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:12:55.709092 9565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:12:55.729450 9565 rpc_server.cc:307] RPC server started. Bound to: 127.18.159.188:43089
I20260926 02:12:55.729470 9631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.159.188:43089 every 8 connection(s)
I20260926 02:12:55.731420 9565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-2/data/info.pb
I20260926 02:12:55.733888 19070 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8FmYUC/build/asan/bin/kudu as pid 9565
I20260926 02:12:55.734088 19070 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8FmYUC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388402166281-19070-0/minicluster-data/master-2/wal/instance
I20260926 02:12:55.734536 19070 external_mini_cluster.cc:1428] Setting key 79a6b2d5b0145faa8e30c946a0741ac5
I20260926 02:12:55.737203 9632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:12:55.748469 9632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35193 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:12:55 dist-test-slave-03rn krb5kdc[9403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.188@KRBTEST.COM
Sep 26 02:12:55 dist-test-slave-03rn krb5kdc[9403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388775, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.188@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
I20260926 02:12:55.765830 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.736368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50898 (local address 127.18.159.188:43089)
0926 02:12:55.740990 (+ 4622us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:55.741018 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:55.741735 (+ 717us) server_negotiation.cc:407] Connection header received
0926 02:12:55.744870 (+ 3135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:55.744906 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:55.745594 (+ 688us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:55.746142 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:55.747556 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.749650 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.751087 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.752398 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.752650 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.757432 (+ 4782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:55.757445 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:55.757463 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:55.757522 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:55.761072 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.761920 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.761928 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.761936 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.762081 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.762510 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.762513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.762515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.763206 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:55.763441 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:55.764019 (+ 578us) server_negotiation.cc:299] Negotiation successful
0926 02:12:55.764529 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4257,"thread_start_us":3938,"threads_started":1}
I20260926 02:12:55.768244 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.751494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.190:35193 (local address 127.0.0.1:52458)
0926 02:12:55.753079 (+ 1585us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.753131 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:55.753318 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:55.753845 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:55.753850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:55.754461 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:55.754705 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:55.754714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.756209 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.756214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:55.757402 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:55.757414 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.758257 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.758264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:55.758494 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.759136 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:55.759163 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:55.761576 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:55.765303 (+ 3727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:55.765312 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:55.765320 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:55.765886 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:55.766351 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:55.766358 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:55.766361 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:55.766509 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:55.767076 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:55.767084 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:55.767280 (+ 196us) client_negotiation.cc:769] Sending connection context
0926 02:12:55.767550 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 02:12:55.767870 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1296,"thread_start_us":1121,"threads_started":1}
I20260926 02:12:55.768935 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.751577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52458 (local address 127.18.159.190:35193)
0926 02:12:55.752615 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:55.752623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:55.753265 (+ 642us) server_negotiation.cc:407] Connection header received
0926 02:12:55.753486 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:55.753520 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:55.753643 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:55.753766 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:55.754878 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.755989 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.757628 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.758036 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.758250 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.761838 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:55.761848 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:55.761856 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:55.761919 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:55.765159 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.766055 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.766064 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.766068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.766226 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.766639 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.766645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.766650 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.766929 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:55.767095 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:55.767719 (+ 624us) server_negotiation.cc:299] Negotiation successful
0926 02:12:55.767954 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":754,"threads_started":1}
I20260926 02:12:55.770488 9632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:12:55.774586 9483 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } attempt: 4
Sep 26 02:12:55 dist-test-slave-03rn krb5kdc[9403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.190@KRBTEST.COM
Sep 26 02:12:55 dist-test-slave-03rn krb5kdc[9403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388775, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.188@KRBTEST.COM for kudu/127.18.159.189@KRBTEST.COM
Sep 26 02:12:55 dist-test-slave-03rn krb5kdc[9403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388774, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.190@KRBTEST.COM for kudu/127.18.159.188@KRBTEST.COM
I20260926 02:12:55.786978 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.771335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47574 (local address 127.18.159.189:41019)
0926 02:12:55.771537 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:55.771543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:55.771564 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:12:55.771623 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:55.771630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:55.771724 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:55.771835 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:55.772608 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.773293 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.774492 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.775120 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.775255 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.779811 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:55.779825 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:55.779828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:55.779865 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:55.783671 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.784507 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.784517 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.784519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.784720 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.785181 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.785186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.785188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.785469 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:55.785631 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:55.786114 (+ 483us) server_negotiation.cc:299] Negotiation successful
0926 02:12:55.786376 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:12:55.787501 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.769690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52460 (local address 127.18.159.190:35193)
0926 02:12:55.769859 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:55.769864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:55.769882 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:12:55.769940 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:55.769943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:55.770009 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:55.770083 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:55.771702 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.772499 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.773813 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.774254 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.774397 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.779485 (+ 5088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:55.779493 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:55.779497 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:55.779541 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:55.783703 (+ 4162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.784670 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.784675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.784678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.784827 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.785271 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.785279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.785284 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.785702 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:55.785979 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:55.786909 (+ 930us) server_negotiation.cc:299] Negotiation successful
0926 02:12:55.787163 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 02:12:55.787854 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.771207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.189:41019 (local address 127.0.0.1:47574)
0926 02:12:55.771379 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.771399 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:55.771491 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:55.771862 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:55.771868 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:55.772315 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:55.772458 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:55.772467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.773443 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.773449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:55.774345 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:55.774356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.775481 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.775489 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:55.775712 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.777011 (+ 1299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:55.777035 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:55.779569 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:55.783839 (+ 4270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:55.783844 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:55.783848 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:55.784362 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:55.784876 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:55.784883 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:55.784887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:55.785047 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:55.785595 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:55.785601 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:55.785785 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 02:12:55.787183 (+ 1398us) client_negotiation.cc:240] Negotiation successful
0926 02:12:55.787561 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 02:12:55.789155 9632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:12:55.792078 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.775654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50908 (local address 127.18.159.188:43089)
0926 02:12:55.775840 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:55.775846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:55.775869 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:12:55.775920 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:55.775927 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:55.776009 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:55.776118 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:55.776881 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.777994 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.779788 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.780418 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.780562 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.784072 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:55.784079 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:55.784083 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:55.784122 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:55.788478 (+ 4356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.789454 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.789460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.789463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.789612 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.790370 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.790374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.790377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.790802 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:55.790963 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:55.791480 (+ 517us) server_negotiation.cc:299] Negotiation successful
0926 02:12:55.791715 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:12:55.792091 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.775215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:43089 (local address 127.0.0.1:50908)
0926 02:12:55.775553 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.775577 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:55.775720 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:55.776148 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:55.776181 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:55.776571 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:55.776734 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:55.776742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.778171 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.778177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:55.779641 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:55.779679 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.780584 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.780590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:55.780814 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.781416 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:55.781433 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:55.783850 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:55.788802 (+ 4952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:55.788810 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:55.788814 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:55.789333 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:55.789773 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:55.789781 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:55.789787 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:55.790209 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:55.790973 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:55.790982 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:55.791189 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 02:12:55.791377 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 02:12:55.791687 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260926 02:12:55.798229 9483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7: Bootstrap starting.
Sep 26 02:12:55 dist-test-slave-03rn krb5kdc[9403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388775, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.188@KRBTEST.COM for kudu/127.18.159.188@KRBTEST.COM
Sep 26 02:12:55 dist-test-slave-03rn krb5kdc[9403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.159.189@KRBTEST.COM
I20260926 02:12:55.802196 9483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:55.803666 9483 log.cc:824] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:55.807366 9483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7: No bootstrap required, opened a new log
I20260926 02:12:55.808734 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.790077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:43089 (local address 127.0.0.1:50920)
0926 02:12:55.790367 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.790388 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:55.790516 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:55.791987 (+ 1471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:55.791992 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:55.793091 (+ 1099us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:55.793322 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:55.793333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.794913 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.794918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:55.796342 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:55.796358 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.797206 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.797213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:55.797391 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.798051 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:55.798073 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:55.800943 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:55.804756 (+ 3813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:55.804764 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:55.804768 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:55.805290 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:55.805756 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:55.805763 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:55.805768 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:55.805922 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:55.807383 (+ 1461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:55.807389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:55.807689 (+ 300us) client_negotiation.cc:769] Sending connection context
0926 02:12:55.807892 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 02:12:55.808342 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260926 02:12:55.809012 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.790015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.18.159.188:43089)
0926 02:12:55.791450 (+ 1435us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:55.791457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:55.791481 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:12:55.791531 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:55.791535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:55.791607 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:55.791733 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:55.793497 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.794762 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.796483 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.797012 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.797108 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.801273 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:55.801282 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:55.801290 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:55.801337 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:55.804573 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.805449 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.805455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.805458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.805608 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.806870 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.806875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.806878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.807240 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:55.807400 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:55.808067 (+ 667us) server_negotiation.cc:299] Negotiation successful
0926 02:12:55.808521 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":168,"server-negotiator.queue_time_us":1292,"thread_start_us":1077,"threads_started":1}
I20260926 02:12:55.810133 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.789877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47584 (local address 127.18.159.189:41019)
0926 02:12:55.790040 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:55.790047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:55.790065 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:12:55.790115 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:55.790120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:55.790208 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:55.790306 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:55.792209 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.793403 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.795198 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.795655 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.795807 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.802066 (+ 6259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:55.802074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:55.802080 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:55.802128 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:55.805916 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.806832 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.806837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.806839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.806978 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.807455 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.807463 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.807466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.807880 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:55.809398 (+ 1518us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:55.809522 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 02:12:55.809831 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:12:55.815200 19070 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:12:55.815178 9632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b: Bootstrap starting.
I20260926 02:12:55.817610 9458 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c2f9dd387af14051b54f0c52700df2b7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52460
I20260926 02:12:55.819449 9632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:55.819308 9483 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9dd387af14051b54f0c52700df2b7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35193 } } peers { permanent_uuid: "606b83010d4346569e3eab64700312d9" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } } peers { permanent_uuid: "8722e59c39f648579c296b291b50718b" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } }
I20260926 02:12:55.819741 9483 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2f9dd387af14051b54f0c52700df2b7, State: Initialized, Role: FOLLOWER
I20260926 02:12:55.820358 9483 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9dd387af14051b54f0c52700df2b7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35193 } } peers { permanent_uuid: "606b83010d4346569e3eab64700312d9" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } } peers { permanent_uuid: "8722e59c39f648579c296b291b50718b" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } }
I20260926 02:12:55.820915 9632 log.cc:824] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:55.821461 9640 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9dd387af14051b54f0c52700df2b7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35193 } } peers { permanent_uuid: "606b83010d4346569e3eab64700312d9" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } } peers { permanent_uuid: "8722e59c39f648579c296b291b50718b" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } } }
I20260926 02:12:55.821893 9640 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:12:55.823117 9483 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:55.823172 9458 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 02:12:55.823410 9458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:55.826627 9458 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9dd387af14051b54f0c52700df2b7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35193 } } peers { permanent_uuid: "606b83010d4346569e3eab64700312d9" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } } peers { permanent_uuid: "8722e59c39f648579c296b291b50718b" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } }
I20260926 02:12:55.828063 9458 leader_election.cc:288] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [CANDIDATE]: Term 1 election: Requested vote from peers 606b83010d4346569e3eab64700312d9 (127.18.159.189:41019), 8722e59c39f648579c296b291b50718b (127.18.159.188:43089)
I20260926 02:12:55.831269 9632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b: No bootstrap required, opened a new log
I20260926 02:12:55.848588 9632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9dd387af14051b54f0c52700df2b7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35193 } } peers { permanent_uuid: "606b83010d4346569e3eab64700312d9" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } } peers { permanent_uuid: "8722e59c39f648579c296b291b50718b" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } }
I20260926 02:12:55.849256 9632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8722e59c39f648579c296b291b50718b, State: Initialized, Role: FOLLOWER
I20260926 02:12:55.849967 9632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9dd387af14051b54f0c52700df2b7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35193 } } peers { permanent_uuid: "606b83010d4346569e3eab64700312d9" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } } peers { permanent_uuid: "8722e59c39f648579c296b291b50718b" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } }
I20260926 02:12:55.854099 9656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9dd387af14051b54f0c52700df2b7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35193 } } peers { permanent_uuid: "606b83010d4346569e3eab64700312d9" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } } peers { permanent_uuid: "8722e59c39f648579c296b291b50718b" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } } }
I20260926 02:12:55.854573 9656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:12:55.855916 9632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:55.855989 9607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2f9dd387af14051b54f0c52700df2b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8722e59c39f648579c296b291b50718b"
I20260926 02:12:55.856617 9607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:55.860936 9607 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2f9dd387af14051b54f0c52700df2b7 in term 1.
I20260926 02:12:55.862046 9433 leader_election.cc:302] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8722e59c39f648579c296b291b50718b, c2f9dd387af14051b54f0c52700df2b7; no voters:
I20260926 02:12:55.862687 9640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:12:55.868040 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.834464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52468 (local address 127.18.159.190:35193)
0926 02:12:55.834664 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:55.834672 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:55.845999 (+ 11327us) server_negotiation.cc:407] Connection header received
0926 02:12:55.846426 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:55.846436 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:55.846537 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:55.846624 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:55.849470 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.850859 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.853418 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.854278 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.854403 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.858888 (+ 4485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:55.858898 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:55.858902 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:55.858977 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:55.863217 (+ 4240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.864639 (+ 1422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.864651 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.864657 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.864889 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.865474 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.865482 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.865488 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.865976 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:55.866367 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:55.867102 (+ 735us) server_negotiation.cc:299] Negotiation successful
0926 02:12:55.867442 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 02:12:55.871017 9437 tablet.cc:2369] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:12:55.875352 9640 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [term 1 LEADER]: Becoming Leader. State: Replica: c2f9dd387af14051b54f0c52700df2b7, State: Running, Role: LEADER
I20260926 02:12:55.876029 9640 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9dd387af14051b54f0c52700df2b7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35193 } } peers { permanent_uuid: "606b83010d4346569e3eab64700312d9" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } } peers { permanent_uuid: "8722e59c39f648579c296b291b50718b" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } }
I20260926 02:12:55.892715 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.836639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50922 (local address 127.18.159.188:43089)
0926 02:12:55.836925 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:55.836934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:55.869602 (+ 32668us) server_negotiation.cc:407] Connection header received
0926 02:12:55.869855 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:55.869864 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:55.869988 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:55.870121 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:55.872313 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.873878 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.875951 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.876773 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.876935 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.880968 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:55.880975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:55.880980 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:55.881020 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:55.885379 (+ 4359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.887004 (+ 1625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.887016 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.887022 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.887370 (+ 348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.888199 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.888204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.888207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.888968 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:55.891141 (+ 2173us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:55.891417 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 02:12:55.891902 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 02:12:55.881232 9565 master_runner.cc:186] Error getting master registration for 127.18.159.190:35193: OK, instance_id { permanent_uuid: "c2f9dd387af14051b54f0c52700df2b7" instance_seqno: 1790388774632061 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:12:55.897480 9565 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:12:55.900668 9557 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } attempt: 5
I20260926 02:12:55.900502 9660 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2f9dd387af14051b54f0c52700df2b7. Latest consensus state: current_term: 1 leader_uuid: "c2f9dd387af14051b54f0c52700df2b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9dd387af14051b54f0c52700df2b7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35193 } } peers { permanent_uuid: "606b83010d4346569e3eab64700312d9" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } } peers { permanent_uuid: "8722e59c39f648579c296b291b50718b" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } } }
I20260926 02:12:55.902910 9660 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [sys.catalog]: This master's current role is: LEADER
W20260926 02:12:55.903777 9671 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:12:55.903905 9671 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 26 02:12:55 dist-test-slave-03rn krb5kdc[9403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388775, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.159.189@KRBTEST.COM for kudu/127.18.159.188@KRBTEST.COM
I20260926 02:12:55.918213 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.846960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.18.159.189:41019)
0926 02:12:55.847191 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:55.847199 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:55.879685 (+ 32486us) server_negotiation.cc:407] Connection header received
0926 02:12:55.881017 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:55.881023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:55.881118 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:55.881194 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:55.883189 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.884424 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.887299 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.888001 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.888105 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.896988 (+ 8883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:55.897000 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:55.897008 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:55.897088 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:55.901576 (+ 4488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.911565 (+ 9989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.911571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.911575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.911777 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.916287 (+ 4510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.916293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.916297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.916769 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:55.917029 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:55.917611 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 02:12:55.917862 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 02:12:55.922032 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.901262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.159.188:43089 (local address 127.0.0.1:50936)
0926 02:12:55.901678 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 02:12:55.901710 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 02:12:55.901811 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:12:55.902700 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:12:55.902706 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:12:55.903216 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:12:55.903507 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:55.903525 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.905965 (+ 2440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.905971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:55.907460 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:12:55.907482 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.909188 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.909195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:12:55.909435 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.910302 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:12:55.910329 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:12:55.913418 (+ 3089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:12:55.918872 (+ 5454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:55.918878 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:55.918884 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:55.919471 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:55.919981 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:12:55.919988 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:12:55.919994 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:12:55.920224 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:12:55.920955 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:12:55.920962 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:12:55.921120 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 02:12:55.921292 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 02:12:55.921615 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275}
I20260926 02:12:55.923825 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:55.901717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50936 (local address 127.18.159.188:43089)
0926 02:12:55.902012 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:55.902021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:55.902073 (+ 52us) server_negotiation.cc:407] Connection header received
0926 02:12:55.902220 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:55.902225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:55.902419 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:55.904058 (+ 1639us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:55.904345 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.905814 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.907728 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:55.908925 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:55.909839 (+ 914us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:55.913707 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:12:55.913715 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:12:55.913719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:12:55.913787 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:12:55.918704 (+ 4917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.919665 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.919671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.919675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.919839 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:12:55.920369 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:12:55.920374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:12:55.920377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:12:55.920802 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:12:55.922888 (+ 2086us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:55.923050 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 02:12:55.923332 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:12:55.913148 9416 master_runner.cc:186] Error getting master registration for 127.18.159.189:41019: OK, instance_id { permanent_uuid: "606b83010d4346569e3eab64700312d9" instance_seqno: 1790388775169249 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:12:55.929494 9416 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:12:55.930260 9557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 606b83010d4346569e3eab64700312d9: Bootstrap starting.
W20260926 02:12:55.930428 9674 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:12:55.930595 9674 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:12:55.930756 9675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:12:55.933313 9675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:12:55.934816 9557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 606b83010d4346569e3eab64700312d9: Neither blocks nor log segments found. Creating new log.
I20260926 02:12:55.937561 9557 log.cc:824] T 00000000000000000000000000000000 P 606b83010d4346569e3eab64700312d9: Log is configured to *not* fsync() on all Append() calls
I20260926 02:12:55.947109 9557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 606b83010d4346569e3eab64700312d9: No bootstrap required, opened a new log
W20260926 02:12:55.961730 9586 tablet.cc:2369] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:12:55.963796 9607 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b [term 1 FOLLOWER]: Refusing update from remote peer c2f9dd387af14051b54f0c52700df2b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:12:55.965505 9660 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8722e59c39f648579c296b291b50718b" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:12:55.968842 9557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 606b83010d4346569e3eab64700312d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9dd387af14051b54f0c52700df2b7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35193 } } peers { permanent_uuid: "606b83010d4346569e3eab64700312d9" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } } peers { permanent_uuid: "8722e59c39f648579c296b291b50718b" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } }
I20260926 02:12:55.969242 9557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 606b83010d4346569e3eab64700312d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606b83010d4346569e3eab64700312d9, State: Initialized, Role: FOLLOWER
I20260926 02:12:55.969712 9557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 606b83010d4346569e3eab64700312d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9dd387af14051b54f0c52700df2b7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35193 } } peers { permanent_uuid: "606b83010d4346569e3eab64700312d9" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } } peers { permanent_uuid: "8722e59c39f648579c296b291b50718b" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } }
I20260926 02:12:55.970646 9679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 606b83010d4346569e3eab64700312d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9dd387af14051b54f0c52700df2b7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35193 } } peers { permanent_uuid: "606b83010d4346569e3eab64700312d9" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } } peers { permanent_uuid: "8722e59c39f648579c296b291b50718b" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } } }
I20260926 02:12:55.971112 9679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 606b83010d4346569e3eab64700312d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:12:55.978052 9557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 606b83010d4346569e3eab64700312d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:12:55.978670 9531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 606b83010d4346569e3eab64700312d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:12:55.978345 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2f9dd387af14051b54f0c52700df2b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "606b83010d4346569e3eab64700312d9"
I20260926 02:12:55.988795 9531 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 606b83010d4346569e3eab64700312d9 [term 1 FOLLOWER]: Refusing update from remote peer c2f9dd387af14051b54f0c52700df2b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:12:55.989749 9656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2f9dd387af14051b54f0c52700df2b7. Latest consensus state: current_term: 1 leader_uuid: "c2f9dd387af14051b54f0c52700df2b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9dd387af14051b54f0c52700df2b7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35193 } } peers { permanent_uuid: "606b83010d4346569e3eab64700312d9" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } } peers { permanent_uuid: "8722e59c39f648579c296b291b50718b" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } } }
I20260926 02:12:55.990093 9656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:12:55.990805 9660 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606b83010d4346569e3eab64700312d9" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:12:55.993012 9532 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 606b83010d4346569e3eab64700312d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2f9dd387af14051b54f0c52700df2b7 in term 1.
I20260926 02:12:55.999691 9677 mvcc.cc:205] Tried to move back new op lower bound from 7333432426328412160 to 7333432425994780672. Current Snapshot: MvccSnapshot[applied={T|T < 7333432426328412160}]
I20260926 02:12:56.005316 9656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2f9dd387af14051b54f0c52700df2b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9dd387af14051b54f0c52700df2b7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35193 } } peers { permanent_uuid: "606b83010d4346569e3eab64700312d9" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } } peers { permanent_uuid: "8722e59c39f648579c296b291b50718b" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } } }
I20260926 02:12:56.005620 9656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:12:56.006551 9680 mvcc.cc:205] Tried to move back new op lower bound from 7333432426328412160 to 7333432425994780672. Current Snapshot: MvccSnapshot[applied={T|T < 7333432426328412160}]
I20260926 02:12:56.018020 9660 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2f9dd387af14051b54f0c52700df2b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9dd387af14051b54f0c52700df2b7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35193 } } peers { permanent_uuid: "606b83010d4346569e3eab64700312d9" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } } peers { permanent_uuid: "8722e59c39f648579c296b291b50718b" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } } }
I20260926 02:12:56.018317 9660 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:56.019610 9675 catalog_manager.cc:1347] Generated new cluster ID: 5103f23efc8945d78090cf8e041e9c60
I20260926 02:12:56.019747 9675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:12:56.040994 9679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 606b83010d4346569e3eab64700312d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2f9dd387af14051b54f0c52700df2b7. Latest consensus state: current_term: 1 leader_uuid: "c2f9dd387af14051b54f0c52700df2b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9dd387af14051b54f0c52700df2b7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35193 } } peers { permanent_uuid: "606b83010d4346569e3eab64700312d9" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } } peers { permanent_uuid: "8722e59c39f648579c296b291b50718b" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } } }
I20260926 02:12:56.041392 9679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 606b83010d4346569e3eab64700312d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:12:56.043011 9640 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2f9dd387af14051b54f0c52700df2b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9dd387af14051b54f0c52700df2b7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35193 } } peers { permanent_uuid: "606b83010d4346569e3eab64700312d9" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } } peers { permanent_uuid: "8722e59c39f648579c296b291b50718b" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } } }
I20260926 02:12:56.043373 9640 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7 [sys.catalog]: This master's current role is: LEADER
I20260926 02:12:56.052685 9679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 606b83010d4346569e3eab64700312d9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2f9dd387af14051b54f0c52700df2b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9dd387af14051b54f0c52700df2b7" member_type: VOTER last_known_addr { host: "127.18.159.190" port: 35193 } } peers { permanent_uuid: "606b83010d4346569e3eab64700312d9" member_type: VOTER last_known_addr { host: "127.18.159.189" port: 41019 } } peers { permanent_uuid: "8722e59c39f648579c296b291b50718b" member_type: VOTER last_known_addr { host: "127.18.159.188" port: 43089 } } }
I20260926 02:12:56.053006 9679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 606b83010d4346569e3eab64700312d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:12:56.056272 9675 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:12:56.066232 9675 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:12:56.089039 9675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2f9dd387af14051b54f0c52700df2b7: Generated new TSK 0
I20260926 02:12:56.090265 9675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:12:56.109331 9698 catalog_manager.cc:1259] Loaded cluster ID: 5103f23efc8945d78090cf8e041e9c60
I20260926 02:12:56.109440 9698 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 606b83010d4346569e3eab64700312d9: loading cluster ID for follower catalog manager: success
I20260926 02:12:56.113373 9698 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 606b83010d4346569e3eab64700312d9: acquiring CA information for follower catalog manager: success
I20260926 02:12:56.116200 9698 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 606b83010d4346569e3eab64700312d9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 02:12:56.238039 9637 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:56.224842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52478 (local address 127.18.159.190:35193)
0926 02:12:56.225015 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:56.225021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:56.235744 (+ 10723us) server_negotiation.cc:407] Connection header received
0926 02:12:56.236318 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:56.236326 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:56.236419 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:56.236496 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:56.237776 (+ 1280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52478: BlockingRecv error: recv got EOF from 127.0.0.1:52478 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 02:12:56.240556 9638 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:56.224848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50952 (local address 127.18.159.188:43089)
0926 02:12:56.225055 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:56.225061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:56.238679 (+ 13618us) server_negotiation.cc:407] Connection header received
0926 02:12:56.238987 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:56.238993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:56.239109 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:56.239182 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:56.240270 (+ 1088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50952: BlockingRecv error: recv got EOF from 127.0.0.1:50952 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 02:12:56.242372 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:56.239460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.18.159.189:41019)
0926 02:12:56.239630 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:56.239637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:56.240239 (+ 602us) server_negotiation.cc:407] Connection header received
0926 02:12:56.240503 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:56.240510 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:56.240605 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:56.240682 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:56.242111 (+ 1429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47604: BlockingRecv error: recv got EOF from 127.0.0.1:47604 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 02:12:56.302611 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:56.282772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47618 (local address 127.18.159.189:41019)
0926 02:12:56.282977 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:56.282984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:56.293999 (+ 11015us) server_negotiation.cc:407] Connection header received
0926 02:12:56.294602 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:56.294609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:56.294873 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:56.295016 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:56.295819 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:56.297097 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:56.299272 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:56.299994 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:56.300104 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:56.300967 (+ 863us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:56.301752 (+ 785us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:56.301882 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:56.302145 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 02:12:56.302252 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 02:12:56.350864 9638 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:56.336569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50968 (local address 127.18.159.188:43089)
0926 02:12:56.336763 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:56.336770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:56.348897 (+ 12127us) server_negotiation.cc:407] Connection header received
0926 02:12:56.349153 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:56.349160 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:56.349299 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:56.349386 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:56.350586 (+ 1200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50968: BlockingRecv error: recv got EOF from 127.0.0.1:50968 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:12:56.406630 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:56.387481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47632 (local address 127.18.159.189:41019)
0926 02:12:56.387765 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:56.387774 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:56.399642 (+ 11868us) server_negotiation.cc:407] Connection header received
0926 02:12:56.399888 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:56.399893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:56.400025 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:56.400146 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:56.400944 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:56.402129 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:56.404261 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:56.404962 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:56.405093 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:56.405727 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:56.405908 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:56.405984 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:56.406215 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 02:12:56.406309 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260926 02:12:56.471865 9638 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:56.456996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50976 (local address 127.18.159.188:43089)
0926 02:12:56.457224 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:56.457230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:56.469695 (+ 12465us) server_negotiation.cc:407] Connection header received
0926 02:12:56.470284 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:56.470291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:56.470392 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:56.470488 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:56.471596 (+ 1108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50976: BlockingRecv error: recv got EOF from 127.0.0.1:50976 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 02:12:56.524324 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:56.505708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47646 (local address 127.18.159.189:41019)
0926 02:12:56.505907 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:56.505916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:56.516884 (+ 10968us) server_negotiation.cc:407] Connection header received
0926 02:12:56.517468 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:56.517476 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:56.517579 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:56.517651 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:56.518376 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:56.519658 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:56.521821 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:56.522536 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:56.522652 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:56.523350 (+ 698us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:56.523506 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:56.523580 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:56.523848 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 02:12:56.523954 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 02:12:56.570293 9638 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:56.557892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.18.159.188:43089)
0926 02:12:56.558100 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:56.558108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:56.568340 (+ 10232us) server_negotiation.cc:407] Connection header received
0926 02:12:56.568835 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:56.568842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:56.568959 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:56.569041 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:56.570024 (+ 983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50984: BlockingRecv error: recv got EOF from 127.0.0.1:50984 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:12:56.640087 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:56.632201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47656 (local address 127.18.159.189:41019)
0926 02:12:56.632397 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:56.632403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:56.633042 (+ 639us) server_negotiation.cc:407] Connection header received
0926 02:12:56.633276 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:56.633283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:56.633409 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:56.633536 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:56.634071 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:56.635329 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:56.637617 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:56.638370 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:56.638477 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:56.639191 (+ 714us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:56.639344 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:56.639402 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:56.639666 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 02:12:56.639768 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 02:12:56.669901 9638 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:56.667799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50988 (local address 127.18.159.188:43089)
0926 02:12:56.668118 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:56.668125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:56.668363 (+ 238us) server_negotiation.cc:407] Connection header received
0926 02:12:56.668610 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:56.668615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:56.668705 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:56.668826 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:56.669659 (+ 833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50988: BlockingRecv error: recv got EOF from 127.0.0.1:50988 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
I20260926 02:12:56.689013 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:56.682351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47670 (local address 127.18.159.189:41019)
0926 02:12:56.682600 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:56.682608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:56.682991 (+ 383us) server_negotiation.cc:407] Connection header received
0926 02:12:56.683201 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:56.683205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:56.683281 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:56.683423 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:56.684502 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:56.685602 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:56.687026 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:56.687525 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:56.687672 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:56.688115 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:56.688300 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:56.688400 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:56.688549 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 02:12:56.688638 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 02:12:56.698043 9638 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:56.696097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51002 (local address 127.18.159.188:43089)
0926 02:12:56.696366 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:56.696372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:56.696469 (+ 97us) server_negotiation.cc:407] Connection header received
0926 02:12:56.696659 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:56.696665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:56.696758 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:56.696874 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:56.697774 (+ 900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51002: BlockingRecv error: recv got EOF from 127.0.0.1:51002 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 02:12:56.726157 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:56.718130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47672 (local address 127.18.159.189:41019)
0926 02:12:56.718479 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:56.718489 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:56.718803 (+ 314us) server_negotiation.cc:407] Connection header received
0926 02:12:56.719054 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:56.719060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:56.719166 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:56.719272 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:56.719928 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:56.721206 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:56.723869 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:56.724666 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:56.724809 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:56.725410 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:56.725603 (+ 193us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:56.725679 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:56.725872 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 02:12:56.725945 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260926 02:12:56.743546 9638 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:56.740999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51014 (local address 127.18.159.188:43089)
0926 02:12:56.741191 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:56.741197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:56.741767 (+ 570us) server_negotiation.cc:407] Connection header received
0926 02:12:56.742060 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:56.742066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:56.742158 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:56.742228 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:56.743307 (+ 1079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51014: BlockingRecv error: recv got EOF from 127.0.0.1:51014 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:12:56.785182 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:56.780006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47674 (local address 127.18.159.189:41019)
0926 02:12:56.780324 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:56.780329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:56.780647 (+ 318us) server_negotiation.cc:407] Connection header received
0926 02:12:56.780822 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:56.780827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:56.780901 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:56.780992 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:56.781450 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:56.782141 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:56.783539 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:56.783989 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:56.784079 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:56.784544 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:56.784663 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:56.784736 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:56.784867 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 02:12:56.784933 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 02:12:56.793336 9638 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:56.791194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51028 (local address 127.18.159.188:43089)
0926 02:12:56.791377 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:56.791381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:56.791734 (+ 353us) server_negotiation.cc:407] Connection header received
0926 02:12:56.791900 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:56.791904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:56.791998 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:56.792092 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:56.793137 (+ 1045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51028: BlockingRecv error: recv got EOF from 127.0.0.1:51028 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:12:56.866988 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:56.861318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47676 (local address 127.18.159.189:41019)
0926 02:12:56.861575 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:56.861584 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:56.861948 (+ 364us) server_negotiation.cc:407] Connection header received
0926 02:12:56.862211 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:56.862215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:56.862304 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:56.862418 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:56.862847 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:56.863614 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:56.865149 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:56.865597 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:56.865808 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:56.866222 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:56.866338 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:56.866439 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:56.866619 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 02:12:56.866715 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260926 02:12:56.875197 9638 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:12:56.873146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51030 (local address 127.18.159.188:43089)
0926 02:12:56.873387 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:56.873394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:56.873766 (+ 372us) server_negotiation.cc:407] Connection header received
0926 02:12:56.873898 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:56.873902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:56.873985 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:56.874130 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:12:56.874942 (+ 812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51030: BlockingRecv error: recv got EOF from 127.0.0.1:51030 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 02:12:56.905869 9671 catalog_manager.cc:1259] Loaded cluster ID: 5103f23efc8945d78090cf8e041e9c60
I20260926 02:12:56.906009 9671 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b: loading cluster ID for follower catalog manager: success
I20260926 02:12:56.908650 9671 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b: acquiring CA information for follower catalog manager: success
I20260926 02:12:56.910540 9671 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8722e59c39f648579c296b291b50718b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:12:57.012652 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:57.006776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47686 (local address 127.18.159.189:41019)
0926 02:12:57.006954 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:57.006958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:57.007352 (+ 394us) server_negotiation.cc:407] Connection header received
0926 02:12:57.007517 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:57.007521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:57.007594 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:57.007686 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:57.008258 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:57.009204 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:57.010789 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:57.011232 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:57.011478 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:57.011836 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:57.011960 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:57.012072 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:57.012271 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 02:12:57.012357 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:12:57.025472 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:57.019205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51040 (local address 127.18.159.188:43089)
0926 02:12:57.019500 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:57.019509 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:57.019765 (+ 256us) server_negotiation.cc:407] Connection header received
0926 02:12:57.019941 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:57.019945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:57.020112 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:57.020299 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:57.020714 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:57.021495 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:57.022899 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:57.023463 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:57.023570 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:57.024124 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:57.024788 (+ 664us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:57.024902 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:57.025035 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 02:12:57.025110 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 02:12:57.043231 9804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:57.032472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52488 (local address 127.18.159.190:35193)
0926 02:12:57.033051 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:57.033060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:57.033908 (+ 848us) server_negotiation.cc:407] Connection header received
0926 02:12:57.034090 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:57.034100 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:57.034379 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:57.034621 (+ 242us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:57.035615 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:57.037318 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:57.040460 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:57.041133 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:57.041277 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:57.041943 (+ 666us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:57.042452 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:57.042572 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:57.042763 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 02:12:57.042866 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"spinlock_wait_cycles":2304,"thread_start_us":362,"threads_started":1}
I20260926 02:12:57.046365 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:57.032622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51050 (local address 127.18.159.188:43089)
0926 02:12:57.032824 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:57.032831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:57.039571 (+ 6740us) server_negotiation.cc:407] Connection header received
0926 02:12:57.039832 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:57.039841 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:57.039976 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:57.040056 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:57.041012 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:57.041989 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:57.044120 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:57.044848 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:57.044985 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:57.045501 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:57.045700 (+ 199us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:57.045777 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:57.046017 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 02:12:57.046112 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:12:57.047209 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:12:57.033647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47694 (local address 127.18.159.189:41019)
0926 02:12:57.033874 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 02:12:57.033883 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:12:57.040484 (+ 6601us) server_negotiation.cc:407] Connection header received
0926 02:12:57.040668 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:12:57.040673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:12:57.040769 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:12:57.040862 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:12:57.041436 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:57.042305 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:57.044511 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:12:57.045235 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:12:57.045373 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:12:57.046287 (+ 914us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:12:57.046431 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:12:57.046533 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 02:12:57.046832 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 02:12:57.046936 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:12:57.048517 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 9416
I20260926 02:12:57.049817 9478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:57.173237 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 9416
W20260926 02:12:57.194877 9584 connection.cc:535] server connection from 127.0.0.1:50908 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 02:12:57.195468 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 9490
I20260926 02:12:57.196720 9552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:57.305032 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 9490
I20260926 02:12:57.325493 19070 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8FmYUC/build/asan/bin/kudu pid 9565
I20260926 02:12:57.326619 9627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:12:57.433749 19070 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8FmYUC/build/asan/bin/kudu with pid 9565
2026-09-26T02:12:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4659 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9705 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (375229 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST